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IMPROVING 
A STUDY PROGRAM 



R&D prpductivity is a com] 



Session 1 



R&D PRODUCTIVITY: . 
AND ITS APPLICATION 



Robert M. Ranftl 
' Corporate Director . 
Enginee cing/^esign Mangigeraent 
Highes Ajj-ccraf t Co, 
Cul rer City,. California^. - 



npllex and elusive subject involving the inter- , 
action of many v^iables. Not only afe the mpre tanj^ible faotors of technical 
performance, time, cost, quality, . reliability , etc./ involved, but also the less 
tangible factors such as work >env^rdnment, motivation, attitude, ett. ^ • 

■ Furthermore, determining the true productivity impacf afr"'f'&'''d effort . 
requires a ^suitable period foir evaluation. Efforts that appear to produce pro- 
ductivity gains in the'^short teirm for example may resul^t in, long-term losses, 
and vice versa. * In^ -addition, shorti-t^^ efficiency at ^the forefront of tech- 
nological innovation may^ be low. But the long-term significahce of" the output 
may be high. In r & d, it is .predoi^itTahtly the long-range impact 4f the out- 
put that is important. 

; An extensive study of r &\d productivity was conducted inl 1973- and 1974 , 
^y tjughes Aircraft Company. Thje goal of, the study was t;o identify useful tech- 
niques for optimizing productivjity in\ an r 6J d enyironiaent . The study encom- 
passed not only traditional research and development efforts, but aiso key in-, 
terfacing activities — contracts, finance, marketing, material, manufacturing, 
^^xipportTv and services. The ptimary thrust of the study was to gather data di-, 
rectly from a broad spectrum o/ knowledgeable individuals — executives, managers, 
~si;ifinlJ-sts, engineers, consultants, anq educ^tors^— with extensive experience in 



'The study .Included s\irvjBy=s£af^ 27 iim^ govemnienta^l, and educational 

organizationa;^ interviews .withT^lS gromin^nt.^ cansult-antsj and a comprehensive 
literature 'search in co^^Stion with 12 ^pajor libraries, government organiza- 
tions, professional associations,, atid other special information, sources.^ Since- 
little matieriai was found specifically on ;r & d productivity , the^taslj was ap- 
proached by researching 'each facet of the subject individually. f Results of the 
two-year study effort were 'summarized, in a\l29-page^ bopk-type teport entitled - 
"R & D Productivity," published 6y HtJ^fies' in December 1974.^ As\ th,e 'Hughes 
study bears outy the specific means- for iraproving>productivity^are bnique to 
eaqh individual and organii^ation; there is t\o'' universal formula.^ The ^primary 
purpose of i;he study report, therefore, is to stimulate readers intG| thinking 
about productivity and subsequently taking action*" to improve both th^ir per- 
sonal productivity and that of their^ particular organizations. \ ^ 



This paper,- summari's^ing the major points of Mr. Ranftrl's presentation! to' t^ 
National Conference on Productivity and Effectiveness in Educational Research 
and Developmei^t , appears here with the permission of Research Manageme^nt 
J ournal , in which it was published in, January 1977. ■ > 

© Industrial Research Institatfe, ],977. " * " , 



.... ... . . . . ■ 

The stucly report in it^ 'entirety , stresses that professional management 
and productixfity are inseparable. The malr^^^th to^ improved productivity is 
through .effective management and .leadership hl^gh operating standards, optimal 
application of "^all resouces, a 'creative envirortment , strong people-orientation, 
effective cotMuhlcationy^ cost consciousness, and simplicity in all operations., 

i . .. ^ o - . . 

Since employees take their cues from management, it is particularly im-, > 
p'ortant that management- clearly convey (1) its feeling of ^he importance of 

, productivity, and (2) its stxs^g desire to see active productivity improvement 
effprts' throughout the^organization. Furthermore, according^to the study, the 

neatest prodiiGtivity improyement resul'ta when management take§ a systems ap- 
proach, emphasizing effective tradeoff decisions within the activities where 
improvement is desired. ' -4, ' , 

' The fol\owit>g mJkiagement agtions/were identified in the study ^s conducive 
to achieving Higli, productivity^:' ' - ' 

Ke*eping the organization "-tbned up"f; always searching for 
more productive ways of doing things 

^ • *Coatinually reappraising approaches to planning, organizing, v 
' staffing, directing, and controlling • ' . 

• Encouraging technological innovation and the us^ .of techno- 
logical aids • \ , * 

' s - • 

^ • Eliitiinating all unnecessary functions; not wasting effort 
trying^to improve ti,he efficiency of parasitic .operations 

\ - — ^ . 

• Managing time carefully; always considering priorities and 
potential gains when making an investment of time , : ^ K 

Contipolling operating funds effectively — the ."p6nny~saved", 
approach , ' 

/ ■ ■ " ^ 

Promptly eliminating unnecessary redundancy 

' . • , - ^ .\ 

Applying work elimination, simplification, and standardiza- 
tion techniques ^ 

• Encouraging healthy competiti*on between ^groups or wi,th other 

^ organizations as a stimulus to improving overall productivity 

Unfortunately,' not all efforts to imp?ove productivity succeed. Reasons, 

for' failure — as noted by study partidipant?s include:- 
* * - "ft 

/ • 

• Lack of sufficient inriovation-'-always doing things *'the same 
ojjfl way." (Methods of operation must be continually questioned 
or productivity will tend to "freeze" at sl- suboptimal level.) 



/ 



• Ineffective implementation of otherwise effectiVie plans for 
improving. productivity ^ 

• Improving productivity of one funqtion at the expense of another. 



Short-term improveipents that prove coufwterproductive in the 
long run • • • , i ' - ^ 

■ \ ■ 

Overanthusiasm or underenthusiasm about improving productivity. 
(It iLs important to maintain proper 'perspective and consis'tent 
pace.) , . ' } 

T({( afford the reader a brief insight inti the rep6rt's contents, a sample 
of the* numerous study findings is included herein 



dne feature of the Hughes study is a set 
the study, the many profess4.oti«ls interviewe 
found most useful in identifying productive 
tions. Although responses were expressed inj 
gently focused on' certain characteristics. 



of productivity proxies. During 
were asked what indicators they 
mployees, managers, an^ organiz^k- . 
ia variety of ways,, they consis-* 
pfhe characteristics are presented 



-^n the(_.study report as (l5 The Profiles of aj Productive Employee, " (2) The j 
profile of a Product'ive Managef and (3) Th 
rion. For the interestVof the reader,* the 1 
tion is also repro4uced herein.' ^ 



A Program for Improving R & JD Productivity * 

After completing its initial study of i 
an internal action-prient^d produc^vity impjrovenkeat 
The primary steps in the program were as folllowsi 



1. Top management launched the prograir 
. to 6,500 key personnel,. Recipients included: 
management and corresponding levels of senil: 
was accompanied by a personal letter from^tcp 
and use. - ' • . 



felt 



2. Four months later, top ^hagement c 
line and staff management personnel ask\ng 
factors within the organization that they 
solici ted' candid responses in each person's 
tirely voluntary,^ and there was no requirement 
The 1,200 OTjnnents received were carefully qtudied, 
a list of 17 counterproductive fadtors. - . 



(]n 



3. Top management conducted, a follow- 
This survey identified the 17 counterpro^uc 
fox solution. The 1,400 comments received ijere cat 
forming a composite list of 70 recommendations 



^ 4. Each of the 17\cqunterprcfductive 
currently being thoroughly studied by a, sen: 
actions have beeii taken, and othets will'fo 
the management ^team drawk on whatever cOmpaijij 



formation on specific action's implemented ifi 
ductivity improvement lett«u^frpm top mana 
who~orfginalTy^ received t^e study report, 
^important to tVk success of the'program. ' 



-3- 



Profile of a Productive Organiza- 
bfileof a Productive Organiza- 



d productivity, Hughes initiated 
program\jtff, January 197 5; 



by distril^u^ng the stud^ report^ 
all levels-^ line and staff 
r technical/^rsonnel. Each copy 
'man^gfement^ encouraging its study 



qj^ucted a sample survey of 2,000 " 
tjhem to' Identify any counterproductive 
warranted attentionVv The request 
own words. ParticipationVras en*- 
to rlespond or to sign replies. 

sorted^ and combined, fcirming 



survey of the same 2,000 personnel, 
ive^ factors and requested suggestions 
fuUy analyzed and combined. 



flic 



tors an( 
r mandg< 
ow. In 
resource 
; being coi 



70 recommendations is 
aent team. A number 



arrying out these actions, 
s it deems necessary. In- 
iunicated by means of pro- 
e 6,500 key personnel • , 
Tiis p^ovl&^s the f eedback_that is so 



gj^ment ^to 
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5. To support the productivity Improvement pT^i^ramj voluntary training 
courses in personal, organizational, and managerial productivity were initiated 
in January 1975. Courses consist of 10 sessions, l-lll hours each, and are 
offered aft^r notmal ,working|hours . To date, 17 such^ courses have been offere^d, 
and^all have been heavily ^subscribed. Of particulat significance, ^both the 
president of Hughes Airctaf t Company 'and t;he assistant general manager devote 
an evening to personally participate in 'every course. SuclTp^rticipation 
incrudes the sharing of their personal management philosophies, the discussion 
of company objex:tives, and the spontaneous answering of questions — all in a 
very relaxed and informal atmosphere. 

• » * 

The formal internal surveys — identifying <:ounterproductive factors and 
related solutions — have been completed; emphasis is now on corrective action. 
However, recognizing 'that productivity 'improvement is a never-ending process, 
top management have requested that, on 'a continuing basis', comments and &ug- 
gestion^^oncerning conipany prbdiictivity be sent directly to them for appro- 
priateTattention. I * ' 

A Backward Glahce ' » ^ * . , / > 

Reflecting on -the Hugl|es productivity program, several factors stand out 
as contributing significantly to. its success . These factors^are identified 
below with the thought that some .of them may be of interest to readers con- 
templating a similar program" within their organizations. ^ 




• A systems ap'proacn was 'taken, treating ail company operations. 



• A low-pfof iley -bensibly paced, professional approach was con- 
sistently pinrsued with emphasis always focused on action. Over- 
night results were never\nticipated, but 'significant long-term 
productivity gain,s are expected.^ 

• Strong,' visible top. management support was always evident through- 
out^ the pro^gram. 

• Care was taken to ensure that productivity ,did not become an end 
in itself, or a'cult. There was never any trace of sensationalism 
or drumbeat,ing, an4^ the word productivity was riot permitted to 
degenerate into a buzz word. 

• Productivity improvement w^^treated as a normal line-management 
functioji, with each organization and each individual encouraged to 
productively perform in position. There were no corporate edicts, 
surveillaripe' staffs, or anything that smacked of a big stiok 

•^approach. - ' • » 

» The study report was distributed well in advance of the two sur- 
veys and, did much Jto stimulate thinking o^ productivity. Parjtici- 
pants* responses were very favorable regarding the usefulness .of 
the study report and the value of the two surveys. 

• , -4- 



• The survey responses were completely candid. Although ther^ 
. ' was no requirement ^o sign their replies, all respondents pre- . 
ferred befng identified. Professionals normally ^want. to be * 
identified for thqir ideas flfnd opinions, and appreciate the 
opportunity to present them to senior management, . 

^ • ^^atly managers, at their own initiative, used the study report 
as a basis for auditing their organization's productivity,. 
Subs.equently, they repeated this process using the 17- counter- . 
productive factors as an audit checklist. 

Looking Ah^ad * , , * 

^ ' t '* 

Hughes considers the resul^^ of its xf & d productivity efforts as very 

worthwhile and enCourag^in^ " However, it is recognized that much more remains 
to be done. A follow-on "productivity stildy is now in progress. In this follow- 
on study, the company hopes to expand its understarvding of the subject and 
develop additional tools for improving productivity in' the r & d work environ- 
ment. In, contrast to the original ^ffoyt, which centered on product;ivitry in 
the aerospace indu^tjry, the follow-on study focuses on prdductivity in cDm- 
mercialiy oriented r & d organizations. * ' - . ^ 

As part of the follow-on study, interfaces have been established with 32 
outside organizations and 15 prominent consultants.^ -The literature, search 
and internal Hugl^es surveys' have continued since the end of the original 
study, and ^will continue throughout' the follow-on effort. Findings of the 
follow-on study will be combined with those of the original effort and sum- 

*marized in a comprehensive' second edition of "R & D Productivity," t^Xanned 

•for publication *"in mid-1977. 

Numerous organizations have become aware of the Hughes study. To date, 
more than 2,700 copies of the study report have be6n requested, by outside organ- 
isations (70% industrial, 25% government/military, 5% educational institutions). 
A number of these organizations have subsequently inaugurated their own pror- 
ductivity improvement programs, signifying that a widespread concern for pro- 
ductivity exists and, more importantly, that positive action is being taken. 
Also, the report is being u^ed as a management training^ text in some companies, 
and several universities have adopted it for upe in cours^ and seminars^. ^ 

* The potential gains from increased productivity for any individual, organ- 
ization, or nation are great. ^ Unfortunately, too many are willing to ^ give ^n 
to inertia of the day rather than take positive steps to improve productivity. 
Fundamentally/ each of us has an inherent responsibility to apply, in the most 
affective manner possible, the resources with wh^ch we are endowed or entrusted. 
We must always seek a better way, rather than just^leave well enough alone. 
Improved productivity requires commitment, ingenuity, action, and perserverence. 
The opportunity is there — what»we do with it depends upon ourselves. - 



Sample Findings 



On a valid definition of productivity ; "Produ^ctivity can be defined as 
the ratio of valuable output to input, i.e. /the efficiency and effectiveness 
with which resources — personnel, materials, machines, facilities, capital — are 
used to produce a valijable output." , ^ . ' * 



. On a viable approach to/evaluating r & d productivity : "There are two 
basic approaches to evaluating productivity: (1) quantitative measurement, 
and (2) qualitative assessment. .Highly structured, repetitive tasks tend to be 
suited for quantitative measurement. On the other hand, creative, abstract, non-- ^ 
repetitive tasks tend to be more suited to qualitative assessment. .'.Inability to 
measure quantitatively should not cause concern; where quantitative, measurement 
is not feasible, qualitative assessment offers -a logical and viable alternative." 

On the critical role of -management ; "The key to productivity improvement 
is management. Kanagement's attitudes, actions, and personal example pervade 
the organization and dire'ctly affect , employee attitudes, motivation, and actions... 
A positive management posture serves as an effective catalyst yfor productive 
activity. . .Only when management gives employees fuU^ attention will\employees 
give 'management full at^:ention. When employees .feel their abilities are^ijespected 
and that they will receive proper recognition foiNtheir efforts, they will norm- 
>aily perform effectively and measure up to the expectations of management, . ^ 
Generally, people will drive themselves if 'they (1) know what is ext)ected, (2) 
thHnk the effort' i9 worthwhile, and (3) feel tfiey will gain through effective 
perfofmance." ^ 

On .productivity -leverage ; *'The overall productivity of an r & d organiza- 
tion is largely determined by the productivity of its management personnel and 
thQ top five percent of its key technical staff — people who deal largely in the 
realm of creative concepts, ideas, and judgment/' 

On organization 'structure ; "A minimum number of management levels consis- 
fent with effective operations is conducive to high productivity. . .When organ- 
izational growth is warranted, it is usually more effectlve^to expand in- a hori-' ^ 
zontal mode." • ^ . 

^ On the impact of effective planning : "Jtiu^fective planning causes ufiijor 
drains on productivity--people x^ot knowing what^te expected, tasks proceeding 
out of^ phase with eac^ other, peripheral activities overproducing or underproducing." 

On competency, level : "if key positions are filled by competent people, other 
competent people will be drawn to the organization. If ,^ ho.wever, setond*-rate 
people are selected for the key ro^le8,• they usually attract third-rate or fourth- 
rate people to positions below them." * * 

On optimal staffing level : ''The optimal staffing level appears to be 90 
percent of apparent needs; individual output seems to drop off both above. and 
b^low this level." • , 

• . -6- ' ^ ^ • 

- 10 ' ^ . 
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^ On mis^mploytnent ; "Success in s^taffing means not only hiring capable peo- 
ple* but providirfg an effective match between individuals and their work assign- 
ments... A key and frequently overlooked factor in low productivity is misemployment 

On unnecessary functions/personnel ; ^ "Unnecessary/parasitic .functions and. 
low pro'ducers not only cre'ate a direct drag on productivity, but they ^re d^- / 
motivating to people who are conscientiously trying to produce 'in the main- 
stream of operations. , .Deadwood shpuld^ pruned., out of an organization 'from 
Che top down, since if. top management tolerates deadwood, all levels will." 

* ' ^ , - ^ V ^ \ . 

On the correlation of individual productivity with personal attributes ; ^ 
"Personal productivity dofes not seem 'to correlate significantly with I.Q., ^ 
creative -ability, ejicellence of 'eduGat^.on' or grades, courses taken since gradu- 
ation, ^umber of patents received, or number of papers written. While these 
factors are significant indicators of other important job^relat^d qualities, 
they do not themselves appear to indicate a personis level of productivity." 

On tech nological obsolescence of personnel ; "Lack of basic Ability seems 

J 

to be a^minor factor in the process of technological obsolescence; conversely, 
lack of stimulation/challenge/motivation appears to be the major factor'... 
Although older- employees, are. more likely to become technologically bjDsolete, 
no consisten,t correlation* was found between age and technological obsolescence. 
I4any scientists and engineers maintain creative output and high performance , 
until age 65 afid beyond, and are considerably more producti^ve than, their you.nger 
coifnterparts." / i , * • * . 



On the cQtrelatio'n of ptoductivity with compensation ; ''Company gains from 
increased productivity should be shared with employees who helped bring about' 
the increased. . .There sh6ul,d be a wide spread of salaries, with producers re- 
warded at the expense of nonproduters." 

7' ■-' -;■ . . • • . • - 



•/ 
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Profile of a Productive Orgaflization 



90 



Organizations as well as individuals 'should be- evaluated, for productivity, ^ 
And the study revealed that managers responsible for r & d ^g|anizations use- 
formally or informally — indicators simlrlar tp^thqse used in ,identifyin|^ pro- 
ductive individuals, 'The most commonly ^psed indicators focus on thellRic cha^if-* " 
• acteristics identified below;' Typit:al observations follow each characteristic^' ^ 

■ - . ^ . ' ' , ^ ^ ... • ■ , 

The organization ^Is effe:^tively staffed and is people oriented , Without 
these qualities it has virtually no chance of achieving J\igh productivity, / ' . 
^ Typical obse-rvations : * ^ * *- ' \ • ^ *" • 
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• 'Has efTedt'ive and respected managements/leadership s*" , ' • / ' 

• Has outstanding personnel in key positions < * 

• Has strong internal resources from wtiich ^o dr-aw ' ^ ; 

• Uses people to the best of their abilities-^matcTies ^ ^ 
the assignment to the. in'dividual* > ^ 

• Provides the piroper' opportunities; assignments, jand - . 
• performance, feedback^ necessary for personal growth' ; - 

• Respects employees a'nd their individual differ-ences 
" • Is sensitive ^nd 'responsive to employees^ concerns ^ 

• Keeps employees informed ' ' * , * , * « ' , ^ • 

• Has dn effective system 0!f recognition and reward for ^ '- ^ % ? ^ 
aotitevement - ^ • . .. ^ '-J • ^ « - * 

The organiza|:ion has high standards . A" reputation 'for high»-^echnical and * ^\ 
managerial ethics is the hallmiark of a productive r & d organizatiorf, Typix:al S 
observations: ' ^ * ^ ^ ^ * ; ♦ *^ ' ^ ■ ^ * • 



• ' Has high standards of operations, and^stresses p'^rsonnel 
; and product integrity *" - . ^ \ ^ 



5 . 

Ik' • -Is perfoBmance/^chedule/cost/quality/reliabillty-conscious; 
ri maintains high standards'of acTiievement in th^se areas ^ • - 

• Maj^ntains justice* and equity irx all- operations 

I. \ • Cqntipually strives'to improve operations. 




, The organisation operates in a sound, competitive manner « Prudent business - 
operation and a readirilSss t6 perform and -compete are necessary for productive 
enterprise. Typical observations:* , . ^ ^ 



• Has. clearly ^established,., worthwhile goals ^ / * ^ ^ 
^ ' •Is prof itable-* ^a strong business sense, '^)rev^ils , • 

• Maintains a '||jalitnced capability which is sufficiently 
•broad in scope to assure stability 

- ^- . ' ' 

• Meets it gpals/pqinmitme^ts — c6nsistently gives timely 
responses — can be coiUnted pn , , 

• Is totally committed ' . ' * 
Lives within budgets . 

• Keeps -costs under control . ^ * . 

• Requires accountability 

. • Maintains a sound business backlog 

• Af fords reasonable ^^organizational growth or^ at leagt 
relative* stability ^ 

• Responds quickly and effectively to emergencies * 

The or^ariization has a creative and productive atmosphere « In the r & d 
environment, factors that are conducive to creativity also tend to stimulate 
productivity. Typical observations: * - ' 

Is dynamic, flexible, adaptive, arid free of stifling controls 

• Encourages innovation 'and the taking of calculated risks 

• Furnishes a continual ^low of meaningful and challenging 
assignments , * ' , 

• Maintains effective communication among colleagues ' ' 

^ • Provides , up-to-date technological aids, equipment, and* 

facilities ^. ^ ' . • 

/ • ' ^ < , . 

The organization has a "can do" attAtude and a high esprit de corps . The 
enthusiasm, dedication^' and teamwork of the people in an organization can neyer 
be underestimated as key factors in achieving optimal productivity. Typical 
observations: ' ^ . - . 

• Employees exhibit a genuine sense of commitment and a * • , 
^ , determination to perform. . ' ^ , ' 

* - * , 

• Employees: iiave a _stong_team spirit—they ehjoy their jobs 
^ drnJ are' proud 6f^ their^coritribution to the team. 



• Team members depend on each other rather tiian compete 
. Xiith each other — they exhibit strong ittterpersonal 

tflist. ' ^ * ' 

♦ 

• , Employees have confidence that management fuliy 
•supports them and itiieir efforts. r . , , ^ 

• The organization exhibits mature confidence in the face 
\of difficult .situations — does^not panic. 

• Employees )iave strong organizational -loyalty. 



' NOTES' . ' . 

1. Original productivity study participant^ Air Force Systeims Command, Arir^ 
Electronics , Command, 'Bell Laboratories, Boefng Aerospace Co., Eastmaji^^dak 
Co., Goddard Spac^ Flight Center, Harvard University, Hughes' Aircraft Co., 
LTV Aerospace Corp., Lockheed Missiles & Space Co., Massachusetts Institute^ 
of Technology^ Miramaf Naval Air ^Station, Motorola, Inc., National Bu^*au 

of Standards, Naval Electronics Laboratory Center, Northwestern University, 
Philco-Ford Corp., Rand-Corp., Rockwell International, St^foitd Research 
Institute, Stanford University, TRW, Inc., Uniyersity of California-Los 
Angeles, University of Michigan, University of Pennsylvania, University of 
Southern California, Xerox Corp'. 

2. Copyright © 1974, Hughes Aircraft Company, All rights Sre reserved. Re- 
production in whole or in part, by any means .whatever, is strictly prohibited 
without written permission .from Hughes Aircraft. Company. """^ ' , 

3. Follow-on productivity study participants ; .Air Foi;ce flight Dynamics Labora- 
tory, Air Force Rocket Propulsion Labora^ry, Aluminum Company of America, Army 
Missile Command, Bethlehem Steel Corp. 'burroughs Corp., California Institute 

of technology, Celanese Corp., Chrysler Corp», Collins Radio Group (Rockwell),/ 
Cornell University, Corhing Glass Works, Dow Chemical Co.,^E.I, du Pont de 
Nemours and Co., Exxon Corp., Federal Aviation Administration, Fiber Indus- 
tries, Inc., Firestone Tire and Rubber Co., General Motors Corp., <5oodyear 
Tire and Rubber Co., HugKes Aircraft Co., Merck and Co., Inc., Minnesot^a Mining 
and Manufacturing Co., National Science Foundation, Naval Air Systems Coinmand, 
New York University^ Princeton University, RCA CorpI, SperryRand Corp., Texaco, 
Inc., United States Steel Corp., United Technologies Corpe?,' University of 
Minnesota. . , * « ' ^ . . , ^ / 

4. Portions of this af tide are extracted from "R & D Productjivity — ^A Key Issue 

by R.M. Ranftl, published in Astronautics & ^Aeronautics , June*1976.» 

» ' — ^ — »^ 



4 



-10- 



Session 2 

THE NORM^OF (RESEARCH) PRODUCTIVITY 
' . VERSUS \ ! ' 
THE SPIRIT OF INQUIRY ' ^ 

John S. Packard 
Program Directoif 
Center for Educational Policy and Management 
University of Oregan 



I want to comment on two notions I have about progress in a research proj- 
ect. The first concerns two conflicting views of research progress'. The sec-, 
ond has tp do with the composition of the research team in this adversary climate. 

, ^ So tJbaf^you will know and I^will be reminded of some of the many limitations 
to my ^thesis, I will say a little about my research experience. I have been 
\ actively, a,lmost exclusively, involved .in educational ^research for the past six 
V years!., kit this time has been spent at' the Center for Educational Policy and 
^-nManag^ment at the University of Oregon. The whole of my research .experience 
during this time has been supported by the National Institute of Education. I 
am limited also in that I have not done much work in development or evaluation. 
However, during this pferiod I have been a member of at least four research proj- 
ects and have served as director, of one project which has lasted for five^^^i^s. 
I serve on various committees that develop or implement research policies in 



the center and the university. 
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The title of my paper — the norm of research productivity^ versus the spirit 
of inquiry — reflects the two dominant, antithetical ways that different people 
think about progress in research. One of these is a rational, investment per- 
spective. It is observed in the ^language and actions' of those who .control, ^df " 
manage or afccount for research funds. I cafll it the norm of research producfei^ty 

The Norm of Research Productivity 

~ ' S 

Where the norm of research productivity is strong, one can observe a number 
of pronounced tendencies. Foremost among these i^ the urgent desire to see that 
the financial investment in a research project is matched by the popular acclaim 
the research products will receive. Adcordingly, the ^-investor o^ten detenhines 
what the products will He before selecting a research team^ - The research budget \ 
is set in advance, and nearly eVery expense is. figured In terms of product costs.* 
Products are supposed to be delivered on a time^ schedule, so the investor can se^ 
that- the work is proceeding in .an orderly fashion. Finally, the investor fre- 
quently inspects the research project to see that proper scientific and budget-", 
ary procedures are followed. 

The norm of productivity is strong and pervasive'. The rational Investment 
approach described above is part of out language and our cultural traditions. 
The norm is applied" almost everywhere and few org^ixizations, except perhaps ' 
churches or prisdns, are exempt. 

. -11- • ' ' • 



The norm of productivity has been improperly applied to ediic^tio^nal re-^ 
search. I say this even though the i\orm is openly embraced by /many educa- ' * . 
tional*, researchers. In order to wofk, researchers often must obtain funds by * 
showing allegience to the *norm. In gaining a cbntract, they often make com-? 
il^Ltments which cannot be fulfilled and agree to employ iechniques that' are 

unsuited to the investigation. Those who fail to honor the norm of research. 

produc<:ivity diminish their chances of. obtaining research funds when they 
compete with others who adhere -to the norm. ^ / 

The Spirit of Inquiry ^ i '* / ? • 

' ■ — 'I ■ I . ' 

The norm of productivity is .antithetical to the critical ingtedient of , 
worthy research.*. The norm df pro4uptivity drives the spirit of inquiry out of 
the research undertaking. ^ Where the norm of 'research ptodudtivity is predominant, 
the term "spirit df inquiify^^ Is regarded^as a euphemism for waste, . inefficiency , 
and skulduggery. For in truth, where^ the. spirit of inquiry reigns, research is 
not tKe-neat, orderly, and timely process xhat warms the heart of the inve^or. 
Edifying research has more, ih common with creative endeavors than with manu- 
'facturing. A research undertaking is typically full of uncertainties and am- 
bijguities to which standard textbook solutions rarely seem applicable. The 
spirit of inquiry encourages one to seek ©ut and come to grips with the imper- 
fections in existing knowledge. Vfe rarely find one best, cle^r-cut way to re- - 
solve these. puzzleinents. The spirit of inquiry is thie essential companion of 
every worthy research undertaking. It must nS%*|be cast out ^by the norm of re-^ 
search productivity. , , * ^ 

\ ■ 

The .spirit of inquiry characterizes the opposing view of research prog- 
ressf It is marked iy an unrelenting, often unsystematic effort to disconfirm 
one's current understanding' in the hope that a superior understanding Will 
emerge « This trait can be observed in the behavior and attitudes of our most 
deserving researcher.s. I 

Researchers recognize and appreciate progress. However,^ to researchers^ 
progress is not. productivity in the. investment sense. Progress does not mean 
fulfilling the specifications of someone's order for ar prearranged product. 
Progress does not mean popular acclaim for one's own pr'odudts. Acclaim, if it 
comes, is an Wexpected side^benef it* of research—not its goal, i^ijgress does 
not entail strict adherence to a predetermined, often arbitrary^ ^inS usually 
incorrect research schedule. . 

To researchers progress means accomplishing various, ddxflcui^ techtjical 
tasks. These include idientifying an area of inquii^ that can be btoxx^M. to 
satisfactory completion, "^nd^ defifftng research concepts; a monumentally under- 
rated and misunderstood taskT, Progress means fashioning an intelligent research 
design and implementing that design both flexibly and reliably. Progress means 
collecting, processing,* storing, and retrieving data. . Progress means making 
sense but of one's observations, and.ji'e^ing able to explain to others the* knowl- 
edge that was discovered and. how it was\x€;vealed. Progress aj-so means experi- 
encing growth in one's technical competence and ability to handle larger areas 
of responsibility. It means sensing the* same growth in one's colleagues. Un- 
der the norm of res^a^ch productivity, these accomplishAnts enjoy only low 
regard. . . \ ^ ^ ^ 
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Where the*spirit of inquiry exists hew knowledge is announced with cogni- 
zance of the precarious basis upon which it rests. Whei^e tlje norm of research 
prod>^ctivity prevails the basis of new kinfowlege is not thouj^l weak unless the 
research products fail to^ return sufficient populat acclainu 

Composition" of the Research Team - ^ . 

Faced with a strong norm of productivity, the spirit of inquiry is not 
sufficient by itself to guarantee research progress > In this regard the^ compo- 
sition of the research team is an import^ant element >^ In my experience, the 
most successful research ,teams were composed of a mixtur.e of esteeioed senior ^ 
personnel who embodied the spirit of inquiry and rathe^r bright junior colleagues 
who held their senior teammates in. high regard. The least successful teams were 
composed exclusively either of junior per^nnel or dominated by persons who 
condupted themselves solely to please or outwit their .investors. 

/ . ^ ' ' . ^ " • 

I have often wondered about these observations. I think the explanation 
is as follows. A spirit of inquiry leads to considerable* debate, .uncertainty 
and indecision. Some of the. most difficult barriers^to research progress are 
the inabilities to. if^solve debates, to proceed under conditions of uncertainty, 
and ta know how and when to reach a decision. Where tije norm of productivity 
was strong, these barriers were insurmountable. However, senior colleagues 
who have the respect of their teammates and who approach difficult issues to ^ 
find their lessons can move ove^ these barriers. Junior colljetagues who find 
intellectual excitement in each new ^tep can proceed with confidence in the 
direction advised by the senior team jnembers . v 

Such a team is unlikely to make false promis*es either about its research 
products or the time and rate at wliich they will be produced. Such .a team is 
likely to enjoy at least the illusion of investor confidence. Institutional 
managers are less likely to confront serious scholars with petj;y complaints. 
Investors ^are unlikely to treat respected figures as if they wer^ students in 
their first reseTarch course. Such % team is also better able to identify and 
balance th^ conflicting demands of the norm of p^roductivity and the spirit 
inquiry. Buch a team is better able to recognize .when and'to- justify why a 
manager's recommendation should be ignored. Finally, such a team is less sensi- 
tive to' the manifold criticisms it receives for not fully honoring the norm of 
research" productivity ; 

*. \ • * 

Conclusion ' • • . 

I am aware that I may h^ve exaggerated the distinc^tions between the spirit 
of inquiry and the norm of productivity 'sopiewhat . I am also' aware that I haye 
described an old and familiar debate. I ba}ce no apologies for these indulgences 
I think it is important to remind ourselves, investors and researcher$ alike, 
that we sacrifice the spirit of inquiry when we are over-committed to the norm 
of research productivity. We cherish the heroes* of science who maintained a 
spirit of inquiry agairfst great odds. Let us maintain the spirit of inquiry in 
our own time. 
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The theme of this conference might be* restated, "How does one get researchers 
to produce more products of higher quality?^' !• think this is, the wrong ques- 
tion, for it implies that Researchers are doing someone else's work. Neither 
productivity, "In the investment sensed not progress, in the research sense, will 
be high upder the conditions. Serious scholars do not suffer long the inanities 
of the norm of research productivity,. They often are replaced by less compe- 
tent personnel vho adhere to the norm of productivity. Work as hard as they 
might and often do, lesser researchers rarely give us anything of merit. On the . 
other hand, if competent people were allowed to exercise the spirit of inquiry . 
fully, a conference such as this^'one would be inconceivable. 



Postscript , I * i ' , 

Although this paper'did not enjdy/ the benefits of formal review at the con- 
ference, it did evoke various sorts of useful criticism. Some participants 
suggested that the paper failed to 4istihgtjiish adequately j^etween educational 
research and educational development. The' Rpint seemed to be that the norm of 
productivity should apply to the latter, but n9t the former. Indeed, Virginia 
Thompson's paper expresses this point^of view. Others reminded me that educa- 
tional researchers often justify their work on the basis of its immediate 
utility for the school setting, In so doing, educational researchers evoke this \ 
control mechanisms that are used to manage educational development, ^fy colleagues 
also pointed qut that the spirit of inquiry lacks an eloquent spokesman and has 
no political currency. In reference to these criticisms Alexander Field'^s ad-* 

relied on the assumption that research productivity should be evaluated in 
teH^s of its impact on schools, Hpwever, Leslie Salmon-Cox was able to show 
that e^ycati^nal r^^search productivity might be evaluated more intrinsically, 

1 Agre^that the educational research, especiaj.ly theory-based research, 
has ofteij beeiKpromoted on its potential for solving practical problems. Hope- 
fully this • tendeh<;y ds weaker than in times past. It has created an additional 
genre of practical^^roblems that researchers cannot solve, I agree that educa- 
tional resiearch lacks'^ eloquent spokesman and political currency, • I doubt 
that the times will pro^Hce either. However, I cannot accept the premise that 
the norm of productivity should surround all of educational development. Rather 
it is my observation that edhcational development also requires the spirit of ' 
inquiry* The norm of productivity might be appropriate^ in the latter stages of 
development where, I presume, the technology becomes highly crystallized. How- 
ever, in the early stages of developmental work knowledge is often quite imper- 
fect. Progress, rather than productivity, should be the dominant perspective. 
To the degree tha.t development ^nd research are 'linked, ^t is often not clear 
when a project has moved from 'one enterprise to the other. In my experience the 
nopn of productivity has been responsible for the lack of progress in development 
as well as research projects. 
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THE RESEARCHER-DEVELOPER DEFINES . • \ 

PRODUCTIVITY AND EFFECTIVENESS " ' • \ 

' David L. Williams • . , 

^ ^ " — " V \ Program Director * , 

Southwest Educational Development 'Laboratory 

Austin, Texas - \ 

% ^ . . . \ . 

o . Quite, of ten* the terras research and development (r & d) are misunderstood^ 
and misused in the field of education. Much of tbe confusion tends to result 
from not having a clear understanding of the purpose of each term\ It coul4 / s» 
be simply stated that the purpose of educational Research activities is to 
attempt to obtain a better understanding of some phenomenon* while the purpose 
of educational development activities is to concep£ualize, /produce and test the 
•viability i)f methods., or strategies seeking to accomplislvspecif ic goals or^ ob- 
jectives under certain conditions. Both educational research and development 
activities are usually undertaken to improve the quality and relevance of edu- 
cational experiences. To help ensure that these activities^ play a significant 
role in enchancing educational experiences, *it is necessary for their purposes 
to have clarity and be easily understood. 

\ti: * Productivity and effectiveness are among the necessary ingredients €or 
successful educational^research and development efforts. Those of us who are, 
involved with and/or manage. such efforts itfust maintain a constant awareness ia 
our work of these two ke^ ingredients. There are several factors which cause 
managers of educational r & d to (1) be concerned about, (2) work toward being 
accountable for, and -(3) attempt to exercise' a measure of control over, with 
respect to productivity and effectiveness. Some of these factors are character- 
istic of each of our educational r & d situations whereas others are^ unique only 
to certain individual situations. . ^ * 

Productivity aud effectiveness are highly *sought after elements in most 
social, economic, political, educational, and even religious endeavors. Pro- 
ductivity essentially refers to the poWer of some phenomenon to* produce, or its 
productiveness. Effectiveness basically refers to the power of a phenomenon to 
produce an effect; a desired effect; being in operation^ active; beiijg impres^^ 
sive or its itnpressiveness. In order to ensure that/^fiese elements are an inte- 
gral part of educational r & d in my work ^ SEDL,-it requires a mustering of 
effort in several* areas. Let me mention these briefly There is no particular 
priority or ranking in the manner in which I discuss the procedures that I at- 
tempt to employ. I shall mention some which probably are familiar to each of 
your efforts, .involving educational r & d. And, I sha^l "mention others which I 
consider to be particularly important to »my work in helping to ensiire ^productiv- 
ity and effectiveness. ) \ 

1.^ Planning the R &^D Effort 7^ 

Thorough planning should precede th^ implementation on r. & d activities 
contitiue throughout these activities and lead into the conduct of future -activi- 
ties.. R&D planning needs to proceed from some getjeral or overall concept, 
to the particular program/project ^area*, to specific individual objectives 



and t^sks* This is necessary to help. ensure productivity and effectiveness. 
Planning for r & ^d must be continuous 4nd flexible enough to ^allow for altern- 
ative courses of Action when necessaryv^ Care must be taken to avoid concen- 
trating the planning effort, only on one area or, phase of r & d activities. 
jB^ch aspect should , be fully, addressed. Planning has tp avoid the notion oft^ 
1^ espoused that "no research/development effort is a failure because you. can learn* 
some^ing from it whether ij: is a failure or a success." This kind of approach 
f^eqWntly leads to haphazard r ^ ^ planning and activities. Thus, planning/ 
miist include establishing clear goals, objective's, procedures, ass^ssmenjjs and* 
desired outcomes in some logical and sequential order which can contribute , to 
productivity and effectiveness in educational r & d. " * .% ♦ 

2, Organizing the R&D Effort 



Making the neces:sary preparations for r & d ^'to \h5^^arried out involves - 
clearly delineating the tasks and responsibilities. This inCliidfes identifying 
what is to be done, who is to do it, and what alternatives or cdntingencies are 
in case something falters or "requires revision. Organization should be ddne 
according^to the tasks to be accomplished rather than^olely by position or 
title. A clear outline or schedule of deadlines and target dates should be in 
the hands of all involved, alpng with a system of reminders. In order for 
r & d to be truly productive knd effective, no task should be-'too great* or* too 
small for those involved. ^ 

Varying the organizational structure, "switching up," can often provide a 
welcome change of pace for r & d staff. Shifts in emphasis and direction wh^ch 
are congruent with individual skills and interests oftjen can stimulate new 
thinking where stagnation may have occurred. Thus, organizing for r & d should 
allow for variation in the deplojrment and utilization of staff capabllit^Les to 
assist with productivity and effectiveness. ^ ^ 

3. Implementing the R Effort 

Actual carrying out of the r &. d effort should be done as closely as possible 
according to the plan and organization for action. The manner in which tasks, 
are initiated and QompleteS will be 'determined ^b/ everyone's comprehension of 
w.hat has ^to be accomplished. Special stress should be placed on the importance 
of each person's work, and how succe^^ful completion of the r & d effort depends 
on the combined contribution of everyone. 

It(g)lementation of the r & d efforts can also be enhaticed' through providing 
staff with a variety of situations in which to^work. A change of scenery could 
be, most condyciye to ^timuiating staffers to be more productive*and effective. 
In addition to providing a variety of working situations, offering the oppor- * ^ 
tunity to work or interact with a variety\of people,* when possible, could con- 
.tribute to increased staff output. The point here is that worjc in r & d efforts 
should not be allowed to become -dull routine. Implementation activities which 
offfr some variation in how they ar6 carried out could be critical to r & d 
profiuctivity and effectiveness. . . 



Keeping t/taek of r & d activities; once begun, is a very important factor 
which contributes to productivity and .effectiveness^ In the monitoring phase, 
several key aspects shpuld'be kept in mind. In this 'f)hase, monitoring. must 

(a) be continuous, i.e., daily, ho*uiJ.y (sometimes) ,. etc. 

(b) be obtrusive and unobtrusive 



(c) be systematic £^nd sometimes unwavering i,n fomiat 



or focus 



(d) be positive in approach s6 as^not tp alienate those 
iavolved in r & d efjfort - * 

/(e) be cleaf inipurpoge and reflect a direct relationship . 
•betweeti these activities ^nd the outcomes bei^g 
sought * ' ^ 

*> 

(f) make' some alloi^ce for "on the spot" analyses and 

decisions which help facilitate activity completion 
t 

^ (g) be conducted as close as. possible to specified tasks, 

, responsibilities, plan of action,, sttedule, due dates, ^ 

5.: Evaluating^ the R&D Effort ' ^ 

Evaluation of the r- & d effo'ft starts with its conception. Analysis and 
-feedback sh9uld occilr constantly throughout the duration of this effort. Every 
attempt should be^made to recognize and deal with problems as soon as thfiy 
aris^e. Assessment of the r & d^effort should basica^Lly* be done according to 
its goals, objectives, ^and expected outcomes. Both internal and external as- 
sessment is necessary. 

.. t ... - 

All critiques and analyses ofv the r & d" effort need to be concise. They 
should provide clear implications or directions for revisions where needed, 
Brief 'but precise Tlarrative descriptions should be maintained concern!^ 
evai^iation of the r & d process and its progress. . 

Specffic Methods^ to Ensure Productivity and Effectiveness . , * 

These five facets of r & d efforts are in no way a comple,te description of 
what occurs in* thes6 kinds of. endeavors. However, they generally i;e^present. what . 
-are important conlsid^rations^when xonducting^ r & d. As for specific methods that 
I emplo^f to help 'ensure ""r & d p^ductivity and ef fectiYetiess ln my work at SEDL, 
the following discussion ,is 6ff«e^. * • 

0 

First, I attempt to devjftlop a sound knowledge base or background fpi; the. 
particula'r r^& d effort beitig undertaken. This is critical to providing ap- 
propriate leadership and direction to ataff when and where needed. I have found 
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that it is advantageous to the program's goals, and to me as a director^ if 
staff can perceive you being as involved as they are in carrying out this 
- effort. In addition, there is kill the subtle pressure on and expectation 
of miniority r & d managers to know and do a little more. As r & d managers, 
minorities, are still not readily, accep^d or respected. Thus,, there has to 
' be a constant pufh*^r acquiring more Knowledge, even more than may be re- 
quired normally. ^ ' ^ 

^ r ■ ■ . • • - 

Special attention is given to developing a knowledge base whicTi represents 
• *what is known arid reveals what Jiew information is needed. The information 
assembled* then comes, from as broad a perspective as possible in' order to guide 
the r & d effort. * . ^ • 

• - 

- Second, I insist on being provided with the needed detai4-s -and documenta- 
tion for r 4 productivity and effectiveness. An attempt is made to empldy 
a "perfectioni^st attitude" toward tfie work to be completed. Displaying this 
kind of attitude do^s not guarantee that perfection will be attained', but the 
infective ^-e^ to' insty.1 motivation, ,to strive for the best r & d effprt 
possible. Such an approach to r & d lead^ irfe to reject vague> or ill-defined 
approaches, plans,' implementaxion and evaluation of the r & d efforts. 
' ' , ' ^ W 
^eing* kjiowledgeable about each step of the r & d prpi$:ess as it takes 
place aiid being involved with decisions for alternatives when necessary are 
critical to improAX^g productivity and effectiveness.- ' This ^requires holding 
staff accountable for their . responsibilities as each phase o,f the r & d effort 
proceod^i^ ,Cidse examination and review of r & d products/strategies ^.at every 
level in th^-p^ocess are qritiq^l to helping ensure r '& d productivity and 
effectiveness. ' * ^ 

^ Third, I, attempt f% transmit^ost of the feedbacU and encou^^ement to 
staff member through informal, individual interactions. When appropriate, 
though, iSJdb praise staff efforts i|i small or large group settings. It is my 
belief that calling personal , attention to accomplishments and areas of addi- 
tional needs is an effective method of increasing pro4uctivity . During inter- 
action witf^ staff members, I alm6st always meet with them on their own "turf" 
'(work area). My percepition is , that they' f ee^l more at ease, secure and impor- 
tant when I come«^o them. " * ^ < 

Staff Tuembers are allowed to, describe and analyze problem situations as 
often as possible, and then take a major role in helplhg-'to resolve them. This 
is an effort to encourage and maintain coopera^ve problem solving rather than 
authoritative decision making. keeping^-^ITaf f informed, to the extent possi- 
ble, of external, and internal'/decisitjns, actions, concerns, frustra,tions,, 
achievements, etc., it is hop e^ that this wMl help them to understand and be 
more willing to positively contribute 'to r & d productivity and effectiveness. 

Fourth and finally,-! attempt to run an "open shop" in our r & d efforts. 
Being easily accessible to staff, listening to their programmatic concerns, 
acting, when possible, upon suggestions and r Commendations, emphasizing the 
avalae of ; individual contributJ.ons to project success, revealing "who I am," 
"what I'm about" and what my expectations are, all represent most of the me.thods 
I empldy to^ncreas^ staff productivity and effectiveness. In addition, overt 
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attempts are made by me to avoid projecting the ValJL knowing" image. Instead, 
I participate as a learner along with being a leader. This also includes 
interacting with Other SEDL colleagues to gain from and utilize where appro- 
priate their r & d Jmowledge and skill^, ' . 

The points that I have mentioned in the precediivg discussion perhaps ^ 
are not thtt itew/pr revealing. Most of you have employed these and many others 
in an attempt to increase r & d productivity and effectiveness.- Given that 
this is true, my reemphasi^ of these points, if for no other purpose, I hope 
has- served as a reminder that there is ,a set of > basic elements which tend to 
hold true when we speak of ensuring productivity a^^^xef fectiveness in educa- 
.tional r & d. Thus, it is Qur collective effj^rts, the-sharing of^ information 
from such efforts, and the efficient utilization of lessons learned from these 
efforts that can serve- to assist all r & d managers to increase their productiv- 
ity and effectiveness^ ' ' • ' - 
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- For each of us, our week has seven da^3^ our day has 16 wakeful hours, 
,more or less. ' It has always amazed me how, so;pe of us seem to accomplish so much 
more in that time than others of us doT, Much of our differences ^^nb doubt, are, 
metabolic — my college roommate used to^fly^out of bed, pour on hiB clothes, in- 
ject his breakfast, and b'e in the midst of his differential equations before I . 
had washed- the sand from myiyes* (On tjie other hand, he fell Into a stupor 
by 8 p.m., when I- was finally picking up a full head of intellectual steam.) 

^ Apart from metabolic factors, certainly many of our interpersonal differences 
in accojgp^ishm^pt are related to varying natural abilities, and differential val- 
ues an6t^%otives. But what I would like to dfscuss today, from my perspective as 
an individual fesearcher employed in a research center setting, are cfertain in- 
centives and constraints that may b^ imposed to assist individual researchers" in 
their goal to be more productive, givdn their biological and ideological makeup. 
Some of theg^' incentivffa^nd constraints are selfrimposed by the individual re- 
searcher; some are ^structural factors, imposed by ^he research organization'. 

Every person operates with a personal set of probleips vthat liiay impede pro- 
ductivity. To the extent that these problems are unique, solutions are generally 
best made individually; to the extent that they are. coimnonly shared, jit may be 
optimal for^ the work group , to be structurally organized to promote ^their solution. 

One problem \sky be, however, that a structural adaptation that ma/ be advan- 
tageous for some members of an organization may* lead to lower productivity by ' 
others. Tlie leadership must then decide whether to ^modify the structural ar- 
rangement, ^change the employment pattern to phase out ''negatively impacted" in- 
dividuals, or,cif possible, to ''individualize" 'the treatment, adjusting the social 
structure to ^ach person's. style and needs. ^ ^ ' ^ 

The problems that most affect me as a researcher are tliree: the problem of 
extra-curricular demands on my time; the n^ed for social support and immediate 
feedback regarding the worth of my product; and the problem of closure — the . 
ability to see a task through' to completion. \ 

The first of these — the personal distractions issue — ^Is one that may or may 
not be widely shared, but which does not really suggest an organizationally man- 
dated solution. My* family and intellectual and recreational, pursuits are as much 
a part of me as my w<5rk. The strategy of segregating these pursuits in time and 
space from my research activities has become more and mote obvious and necessary. 
Yet this solution admittedly does not maximize research productivity since, it 
would always be possible to extend the hours devoted to research at the expense 
of satisfaction In other realms. But, to compromise *^hese^ conflicting interests. 



I have »elf-imposed a 9:00 to. 5:30 five-day week--something I swore I would 
never do — placing only limited additional time demands 'on my evening and weekr 
•end hours. 

The organizational role is much larger with respect to the. other two pi^ob- 
lems I mentioned — the "need for sdcial support and ob'Jective feedbac?k*and the^ 
need for both pressure and ' freedom to complete a task. These factdrs are very 
m>ich tied up with onje another, ^ - - ^ 

For.,inost peoplB, productive work require^ a mixture of independence from 
supervision an^ from organized Memands' on one's time, oi\ the one hand, and 
collegial and supervisory attentioji, criticism, and emotioi;ial Support on the * 
other, • Too many obfigations on one's time are distjsacting; too much supervi- 
sion can be inefficient; and yet too much independence prevents early detection 
of ei^rors and produces too many points where wQJtfc is stymied by a particularly 
perplexing decision or by the need for aWoutsWer'3 perspective. As Arthur r 
Stinchcombe once commented, "The conviction of intrinsic value of ;ft>rk^ is a weak 
reed to sustain a year or two of drudgery," ^' ^ 

One of the charac'teristics of social science reseatch, particularly bas'ic 
reseafcirj is that the market for our product depends on a long^chain of actions 
beyond our:. individual control: peer review, dissemination, interpretation, and 
so on. We often lack immediate feedback on the overall direction of our activ- 
ities, the quality of our assumptions analytic technique, expository 'ist/le, 
literature coverage, and so on, i . " 

It is not only a concern for objective feedback that motivates an interest 
in collegial and supervisory confcaojj^rbttt ego support as well. For most people 
self-esteem needs to be frequently refueled* Sincerely expressed ihterest in 
what the researchers are doing, sympathy for their proljJ.ems, and sincere' praise 
for what they feel are breakthroughs they have made are bouij^ to encourage fur- 
ther productive activi\y, Thi^ is particularly true if such' support isexpressed 
by someone heid in high esteem,- While this may* be a colleague, it is probably 
more effective when expressed by ^a superior* I would maintain, as a result, th^t 
organisational leadjership that provicies frequent, even daily positivi' feedback, 
even to a degreed and fully professional research staff, is likely to be effec- 
tive in stimulating the productivity of its workers* 

In a num^r of ways our center provides for the kind of feedback an<^suppOrt 
I am talking about.' The center's director for the most part stays very much on 
tx>p of what each researcher is doing. Helpful in tlj^g^regard, I might note, is 
the placement of his office. Instead of being located at the rear of a well-* 
protected outier office of secretaries and assistants, it is strategically located 
(with. wide-open broad double doors as its entrance) at the itnmediatQ left of 
the main entrance to the center — and adjacent to the staff mail boxes as well* 
This location and openness encourages frequeflt communication arid' interaction be- 
tween the director and the professional pfjogram personnel. 

Accomplishing a task requires more than social ^up^ort, of course* The 
stick is often as effective a tactic as the carrot. But there are gentle, ways' 
of constraining inefficiency.^ For many people, including myseljE, havittg "a ^ 
deadline is important. One reason deadlines are important is that our priorities 



^ tend to be reallocated iii the direction Qf sche'duled activities, regardless of 
tlieir intrinsic importance, c Appoiapprents, .classes ,to_.te^h, eating; and going 
home to family ^11 seem to take pfece^ence over getting: research dpne,^ when- 
deadlines are absent. ' ' ' - * ' * ' * . 

Deadlines, however, are useful mainly when supported by Expectations of 
others: conference sessi^^chairpersons or government, civil servants^^ for ex- 
^ ample. For its part, tl>e work setting may impose deadlines in a useful and 
^*nonarbitrary way. Weefely luhohtime seminars which givfe coworkers an opportunity 

to learn about^ york they may not be personally involved, in also provide a g6al 
• to \)e ignored only at the risk of shame for a poor performance: 

Giving social support to colleagues and att6nditig seminar presentations^ 
may .be advantageous tQ the recipient of this attention, out they also constfcL- , ^ 
tute potentially nonproductive demands on the time of, the*^attention-giver. . One ^ 
rpadblock to pro^Sct coinpletton is the prb3.if^^tibn of aiich demands, one's 
,<s<ime. Reviewing one's priorities frequently, and then imposing deadlines Biased. 
opxChose piriorities, will help a peifsoo decline many of these external demands. 
If you have determfned t|iat completing" chapter one is of ^igh priority, and 
fixed it on your agenda for .the day, it is easier to say no to other demands ^ 
on your tJ^e. • ^ 
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Another interference with a well-planned^ schedule. is the demands of your 
^wn assistants. It is e^sy to fall into the trap of adjusting your priorities * 
in order to meet the appetites of secretaries and research assistants' for ad- 
ditional work. On the pne hand, these appetites demand th'at the researchefr make 
some measurable progress — that the code book be completed so that the secretary 
can have something to type— *ut often the question of what to do next is decided 
on the basis of which assistant Is most in need of work, rather than which task 
is most important' to be accomplished. At our center, the organization of support 
systems helps solve this perennial problem. While specific research a^sist^ts ' 
and student aides tend to be allocated to individual researchers (and tJney are 
^<f.nerall3(^ recruited on an individual basis), the services of the secretarial 
staff and^the computer programming specialists are shared by all professionals, 
with p-riorities pretty much based on need. \ ' . " 

There are, of course, other variables involved in the task 'completion prob- 
lem besides social 'support , feedback, freedom frOm interferenjce, iitiposing dead- 
lines, and keeping priorities straight^ '*X 

^One factor in raising the probability, of prbduct completion was suggested . 
by Stinchcbmte — having several proj^^rts on the fire at once. It is inevitable 
.that any given project will' reach- an occasional rut. If one's activities are 
totally engaged by that prpject, such a derailment may make one reflect on one's 
ability io get anything accomplished (or as Stinchcombe suggests, one's en^re 
"Identity" may be held, responsibfe) : Py being able to turn to other activities 
and make progfress in other *^irec£lons, the problem o;a the first project remaii^s 
less generalized an4 the attack on it remains more focused. 

; ' Another incentive is to be^.^ assured that there will be some product £tt the 
end of the tunnel, .One way of a'ccqmplishing thig is for the offeanization to 
£^oduce^ and externally disseminate, a publication series of articles and reports 
by staff members. Such a publication series has the additional ad^santage of 



providing the potential for broader feedback regarding research ideas and find- 
ings beyond the formative stage but prior to submission to professional journals. 
While not formally refereed, papers submitt^^ through such a series will/Rkve 
gone through a certain degree of review and comment from colleagues within the 
organization. At our center, in addition, the articles submitted are reviev^ed. 
by .an editorial specialist. 

For the most part, I have been discussing productivity with respect to the 
taisk accomplishment problem — how to get a given product completed once the deci- 
sion has been made to produce it. Probably more important to overall productiv- 
ity, however, is the issue of task selection — the choice of.whiph products to 
produce. ; • ' ^ - z'*^ 

Questions about task selection are primarily of concern to the managers of 
r & d settings. But individual researchers see these issues as relevant to their 
own productivity as well. While researchers generally appreciate guidance and 
social support in their activities, I' would maintain that, if the recruitment 
task has been properly accomplished, the researchers hired should be capable of 
being responsible for tlfeir own selection of research problems within the broad 
framework* of the goals of -the research center.^ 

Peter Rossi, in a recent article in Educational Researcher , suggested tKat 
the optimum leadership style within an r & d center was not "discipleship but 
the "institutional person." This is 'a rather decentralized style that tolerates 
diversity in the style and content of research activities; where the leadership 
derives personal satisfaction from the independent accomplishments of others 
which bring rewards to the institution and not personally to- the leacdership. 

On the other hand, Rossi siiggested that a consortium of independent .re- 
search entrepreneurs is probably not as productive in the long run as a place 
where the leadership is able to exe^t* rather persuasive influence over the di- 
rection of research activities. " . 

The solution here, it seems to me, lies in a blending, of these leadership 
styles. There should be a commitment and even an expectation that researchers ' 
develop their own specific research plans, consistent both with their own In- 
trinsic disciplinary interests and with the overall organizational direction. 
At the same time, the leadership yould be' responsible to assur^ that such re- 
search ideas, through collegial assistance and support, be elaborated into .re- 
fined, potentially productive research propc^sals^* 

Ther^.axe,' ^df^ course, many other asp^icts ,of Hhe productivitiy issue. There 
a^re definitional problemp such as whose stattdards of productivity oup efforts * 
should be directed towards: the funding agent's interest iti project completion, 
regardless of .value; or the researcher's own immediate valuations of the relative 
worth of different efforts, regardliess of hi^ or her prior perceptions and prior 
commitments to complete projects; ♦ , . 



Also, there are managerial issues such as whether it is best to invest 
manpower in data collection activities which tend to be highly dpisodic and 



demanding of time and effort; or whether to entrust , these activities, which 
~ Tieyertheless require ^extiiejae attention tO;de£aiJ," to aft external organization 
that may be insufficiently motivated to provide the careful oversight required^ 



" '.But these latter are primarily problems of managers*^of research facilities. 
I have instead focused my comment^ on those njost relevant to my own activities, 
a3 a professional sociologist employed in a gtoup research setting » 



V 
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A DEVELOPER'S LOOK AT EDUCATIONAL 
4 . . PRODUCTIVITY AND EFFECTIVENESS , 

^ Virginia L. Tfiompson 
. Program Director 
Northwest Regional Educational Laboratory , , . * 

Portland, Oregon . , ' * 

A "National Conference oh Productivity and Effectiveness in Educational 
R & D" is impressivel My first reactiorf to the title"^was^, "Will .the conference 
bfe' productive and effective, and if so, how?" My anxieties were relieved some 
when I learned that presenters would include not only those of us from educa- 
tional, r & d but also individuals from the business sector of r & d. 'It isn'*t 
that educators talking to educators will not be productive or effective, but 
I do think sometimes, through laak of a profit incentive, educators are not as 
consclous^as we ^hpuld be of "making the most of the buck" in our research and. 
development work. . • , ^ 

> 

Another reaction I had to our conference title was fear tHat* the 'terms, 
might lack universal definitions, which is always a good \^y to tie up discus- 
sion* By the time the terms are defined, Wejre left with no time to discuss 
issues. Webster 's gives us some common definitions of what it means to-be 
"productive": having the .quality or power^of producing, especially iu abun- 
dance; ef f ective bringing about; yielding or furnishing results', benefits 
or profits; yielding or devoted to^^the satisfaction of wants. Productivity in 
my life span has had many meanings and measures: * 

• When I was a tfeenager in the strawberry field, it was defined 
as the number of hallock^ that constituted a crate and for 

^ . which I was paid $.08 per pound.' 

• As a beginning secretary assigned- to type stencils (In those 
days high speed copiers were unheard- of), it meant the number 

• of stencils completed In a day. 

• As a grocery clerk, productivity was measured by the total 
amount of sales on the cash register at the end of the ' ' 
day. . t 

• / As a keypunch"' operator at a large Pottl^hd bank, it was an , ^ 

hourly item , count *on which pay Traisias depended, determined * * 
by the number of cards punched each hour — minus two for « 
error njade. . • ^ ,i ' " ' , 

• There waS'. even a time when my productivity was judged by the 
- number of children I. had. 

v 

•Effectiveness, by definition, implies proven capability based on productive- 
ness in operatiotj and especially st^ss^s ability to perform well and economically, 



Inherent in such performances are the absence of wasted time, energy or material; 
the demonstration of skillful management of means; and technical expertness 
suggested by the term "know how." 

Wi,th a work-ethic, prcjduction-oriented background, I journeyed from teaching 
into J: he .halloaed r & d community determined to make the federal dollar invested 
In our project pay off in terms of usefulness to local educators ^In every pos- 
sible way. This motivation wasn't entirely my own"; it was conditioned in part 
by the feelings of my fellow teacher,s that somehoW educational r & d was a 
dollar waster, seldom yielding practical results for the field. 

For the purposes of this discussidn today, I am a developer. Based on my 
experience'at the laboratory in that role, I feel there are some critical .ele- 
mettts necessary to ensuring productivity and effectiveness in r & d. 

I believe there are some personal. characteristics- necessary to being "pro- 
ductive and effective as an educational developer. First ^ou have. to believe in 
what you're involved in. It takes creative energy, enthusiasm, excitement and 
belief that the impossible just might work. On the other hand, it' also re'quires 
a willingness to have your theories, concepts, and creative work laj.d bare to 
the cold scrutiny of evaluators who might find the project or program to be' 
ineffective, producing results just the opposite of those hypothesized and sent" 
Back to the mental drawing bdards for replanning. Developers have also been 
described as individuals who are intuitive, artistic and craftsman-like in their 
work. Educational r & d, I believe, requires an adaptive, resilient ^risk-taking 
individual > one who is as open to scrutiny as the stencil typist or the straw- 
berry pick^sr. • ' \ 

Insofar as it is possible, I am of^ the^opinion that programs should be 
functiorfal while go^ng through developmental ahd evaluatidn cycles. Some pro- 
grams .are developed in a theorist's study, put oiit^in the world for a test'-- 
while hoping f^or high fidelity, evaluated and then gulled back toithe sttjdy for 
revisions. I believe that the "hot house" development of programs should be as 
short as possible with operation occurring during final development. Ongoing 
cycles of 'development , evaluation, redesign and further development, and refine- 
ment ensure that the Investment will have pay-off more i^apiHly than* if the de- 
velopmenjt happens in an isolated theory environment for an extended period of 
time. Ttn.s process of development and operation is\risk taking and demands hard, 
fionest scrutiny of effectiveness, includlAg JsucceSaes and failures. It requires 
backing off from pet theories (often in pub|ic view) in the light of proof thab 
a concept doesn't work and acceptance of. e^luation findings that a different , 
strategy is heeded^. But for my money, this is * one way ^to ensure effectiveness. 

It goes without saying, I suppose, . that field-based development is of'ten 
inore> messy and time consuming. But, according toithe Rand study, the stud-y of 
NDN and others,,, it appears to be about the only way to 'ensure effectiveness. 
An, issue which needs to be raised also is the necessity for educatiooal develop- 
ment to seek a balance between productiyity and effectiveness. Some observations 
^how that frequently, when development extends over a long period of ^ime, the 
pirogram actually goe^s through radical shifts and thit, at a .point in time, de- 
velopers have to determine the product, under development conditions, is^ finished 

- " ' ; - --26- , 



/ . ' ' ' 

There always i? the possibility that once pilot tests are begun and market^ 
ability examined, changes will occur. Bilt there i§ a need £o strive to bal-7 
ance productivity with effectiveness. 

A developed project, program or product should be moved from the demon-, 
stration test site to pilot site in. as short a time frame as possible'. The 
use. of a limited number of pilot sites to test the program in new environments 
assures that the doj|^lar investment will have benefit for more schoolls and more, 
students.* It tests the stability of theories in new environments and, when 
successful, creates needed advocates for your product. If developers wait too 
long. to begin moving a product' out, the world loses, interest.' Also, if it 
appears; that fhe results of the r & d efforts will never be available to other 
users ,v an atmosphere of hostility and mistrust 'can ev*n develop. 

R&D projects and programs should recognize, plan, and staff for changing 
program emphasis*,' or what I call the .developmental cycle. During different 
phases pf this cycle, ^different kinds of expertise .can "save" the program and 
he^lp it to the next cycle. These^ phases are: 

• The development phase itself: At this time you have creative 
developers, seasoned practitioners, creating and beginning ^to 
implement your design. They are the risk takers, the "we can 
conqueX the world" troops. There needs to be a word of caution 
here. Too often we make our development too people dependent. 
That is, the- developers build themselves into the tfFojduct so ;* 
tightly that we cannot effectively diffuse -without those people.* 

During the total cycle, but especially within a year or year and 
a half, yojir project will need the systematic ,t critics^l, strong^ - 
evaluation perspective. The results of these evaluat^ions^ must be 
realistically send honestly reported — for both internal changes *and 
external audits. . A good evaluatioti team can, many times,' 'save a ^ ' 
program for further funding and brighter futures. . ' 

• Somewhere in th^ second year of operation it becomes critical 
to gather together and synthesize materials and document how 
a program or project works.^ At first it may not be clear or 
you may jiot have an agreement on what will be done with this 
material j' but it is critical that the program and its develop- 

t ment be described, I believe, while some of the priginal de- 
velopers are around. It is^ equally important to'document, for 
future users, what didn^t wgrk and why — as it is t;o> rSt>ort what 
worked and how. * * ' 

\ • Lastly; it is critical to .plan for the spread- of" information 
and procedures for transferring the capability to install a 
• program. And this is\^here the cycle of training and' technical - . 
assistance becomes import;aut. 'This cycle must be well planned 
and occur early in the development cycle to insure the grefttest 
spread and availability for* the least amount of money 



, V, 

Given that I believe there 'is a developmental cycle, I believe that good 
management of educational r & d should make maximum uSerof personnel sLn these 
various cycles. For example, an individual who has been part of the develop- 
mental teaip can, through training new staff to assume their roles in the 
program,' help develop training materials for handbooks and program installa- 
tion. . Once the program is operatlo^air-that developer might, then become a 
trainer of *staf f for new programs. Likewise, an Individual who has been 
heavily involved in documentation of .th3 project might become an effective 
trainer— taking with him/herself an entirely different perspective because 
of a rather analytical look .^t the program or project. .* 

Educational r & d program ould be made arvailable to as broad an audir 
ence as possible through as manyVx?^"®^ possible. » Some developers have 
such a personal, vested interest in th^ir project that they are reluctant_to 
let it, go for fear that it will be eitl^er -misused or used inappropriately. ^ To 
me the reverse is true. If I believe strongly enough in the program or project 
and its^ potential for schobl improvement then I also believe that.- the ways 
adopters or adapters choose to use it ^4.11 also be beneficial to them, perhaps 
not in the way I had originally intended, but maybe even in a; better and more 
productive way. Spread of an innovation should occur In a multitude of ways: 

• Making potential users aware of the product or program through 
brochures, multi-media presentations,- articles , word of mouth. 

• Taking appropriate parts of a ptogram and packaging them for 

use in classroom or local educational settings with no training * 
or technical assistance required. 

• Providing short-term 'training sessions in some of\h(!B core aspects 
of a program, 

• Producing step-by-step handbooks which allow potential adopters/ ^ 
adapters to take the developed program and implement it in their 
own setting with no training or technical assistance — if that is 

. the way>they choose to proceed. i ' . 



•* Des/gnlng and p^oyfaing training to install a total program on as 
cost-effective a basis as ^possible. * 

' / ; . ^ ' * • 

^ V, Taking a developed program or project one step further by suggesting 
\ ' creative ways* strategies developecl in a program can be adapted and 
' used to, meet different needs. My opinion is that wl^ole model spread 
X alone is nonproductive jt^id frequently ineffective. ' 

If they believe In their program or project, developers ought to operate on 
th;^ asiSumption that the current year is the last year of funding (and well 'it 
mlg|it, bq) . If they believe strongly enoagb in their own program an^ its po- 
t^^t^^l effectiveness as an agent of change, they will want the program to last 
/ ^95^et^^J^^^^ the., funding does or not. Wiph this kind of pressure, there 

' ^■'i^r^^^ u^g^c^to get products oat, to. get people trained, to build sup-^ , 

j)(?^tv;i|eS^rks^-^ new Tnaterials deve^pped so that the program is lasting. 



My experience with tUis type of urgency is that it forces developers* to be 
creative^ productive and strive foi an effectiveness that is characterized by 
little wasted time, energy, or materials* And that in itself demonstrates 
skillful management of means and technical expertise. 

y ' ' 

For those of you in executive positions, I believe it is imperative -for 
developers in labs and centers to have a sense of the mission of your organiza- 
tion. Our mission is: t . ^ 



\ . ^ ' 

• developing' and disseminating effective educational products 

• ' conducting research on educational problems 

t providing technical assistance in educational problem 
solving 

• evaluating effectiveness of* educational programs and 
projects 

• providing training in educational planning, management and 
instruction 

• serving a^ aji information resource on effective educational 
pr-ograms and processes. - " ^ 

Good management encourages study on a continuous basis of development proces- 
ses and theory. Sometimes even. impatience may also be a necessary executive and 
management* trait — to constantly ask, "can we achieve the samesvpr better quality 
in a shorter period of time?" Freqaently, under the federal fmding arrangements, 
there^ is little, incentive for speed — rather the* incentives to bt slow and expen- 
sive are high. \^"fKfere is frequently no bonus for f inishiiigearJhJj^ Management 
also has to be aware of the amount of time develo^ers^^ve to^'spend in writing 
reports, proposals*, and meeting t^eir federal dematlSs. From my experience, there 
are individuals in executive and management^ posiliions who allow~no, who encour- 
age developers to be^ responsibe to clients in tneir region; to be creative in the 



development of p^ogr^ins and projects; £md- to be 
execution of their work. We, as developers, car 



focus on today id what we must ask — require of * ourselves. 



productive and effective in the 
ask no more of others. What I 
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The most valuable resouce available to any organized group, whether it be 
society, a company, or an institution, is the skill, 'intelligence, and produp- 
tivity of the indfvidual', fhis 'has been aptly tertaed "huanan resources/' The 
principal role or concern of^ management, then, is to realize the £uil potential 
of each individual to produce, innovate, and create in a manner that will In- 
terBase averall productivity, profit, or contribution "to objectives. Although 
\we are concerned primarily with research and development, ^and thus, with sciea- 
Cfists and technol^sts, the author believes that the successful management of 
^11 people involves. the same principles.* 

An appropriate "management theme" or operational context can be 4-clenti-^ 
2d by examing history'. 'Since history documents the results, of organized hu- 
is* efforts under a variety qf circtamstances, it must also tell us what con-:^ 
may be appropriate for a group of researchers. Consider the following: 

1. Toynbee's study of history, concludes that societries rise and 
fall on whether or not they rise to meet external challenges. 



\^2. One of Western marf's most creative milestones was the 'Renaissance. 
It is believedNrl4t >this resulted f tom^ the competition and con- 
flicts between the Italian states of that period. 

. . . / 

3. Popular wars ha,ve historically been periods of Inventiveness and , 
\ productivity. . ^ ' 

The fact that the "human dimension" is* important to productivity in r &. d 
is apparent. Economic" studies of the impact. pf research and development on 
corporate Rrof its and growth show that correlations^ of r & d expenditures versus 
size and growth, r & d dollars versus net profits, and size of research gi;oups 
versus their contribution to new products, are inconsistent.^ Some companies are 
more successful than others, and some groups are more productive than'others. 
Such .studies, by th^r very natifre, must eliminate -or reduce to a minimum the ^■ 
humaa part of the equation. Is it not possible, then, that in "the more suc- 
cessful cases," management has been more effective in establishing a more crea- ^ 
tive and^ stimulating fenvirohment? ' , ^ 

Based on some understanding of history and personal observation by the author 
this discussion presents a general management theme or context and some ele- 
ments in it^^ implementation. s ^ 



Basic Manageme n t Theme or Context ' * • • 

: — : c . . . 

Following the example of historical "success" and the natural responses of 
Western man, -the basic management context for increased productivity and crea- 
tivity in r & d consists oft (1) challenge, (2) response, and (3) reward. 

Outstanding management^, then, involves initiating and closing this cycle 
of evfents. Unfortunately, the* m^ny factors involved in this cycle complicate it 
beyond the simple expression given. Since we are dealing with people, building 
and sustaining it becomes an art. Operating within specific, narrowly defined 
rules does not work because people differ. 

Challenge is the first stfep management shou-ld take to create an atmosphere 
favorable to productlvityycrfeativity ^ The chairenge shoulc ^be tr ansmitted to 
all members of # the or^nj-zation and in different forms since, peo^e and their 
functions within the organization vary. For som'fe individuals the challenge 
comes from within. Cr^eativity should be cdnsidered in relation to not only 
scientific activities, but also sales-, ^adnvinistration, and services, all of 
which are functions of an organized feroup. Thus^jwe are challenging the staff ^ 
to innovate, suggest changes, create , new products, and develop new technology — 
all aimed at improving the oatput the organized effort. This is a. continuous 
activity that takes time and effort. 

Definition of the challe.nge is, perhaps,, one of the mo$t difficult parts 
of the creative process. In most cases in today'' s complex technqlogy, problem 
definition and/or challenge formulation requires the combined thinking of a num- 
ber of researchers versed' in different disci})lines . Managemep.t can help £o ini- 
tiate and guide the creative process by implementing the below-characterized 
approach: * . * . " 

1. Indoctrinating the staff with a^reason^lie understanding of 
the of-ganization's objectives atid how the people taight con- 
tribute to them. *. ^ 

2. Presenting challenges to ,tJle entire staff. 

3. ' Communicating the problems and/or challenges clearly enough 

to minimize confusion of the individual. 

4. Characterizing thp challenge, whenever possible, in the form \ 
of ' a tasb and/or a goal or purpose, with appropriate freedom 

, to act and authority t'cJ implement. ' ^ . ' 

' . ■• * i • 

( • ' 

5. Creating an environment in which ^searchers can communicate 

and work with their counterparts in different parts of the 
^ ' organization without impedaftce from management. 



6. Providing frequent changes of challenges and new ^t^fpcs rather 
than permitting sustained activity on one project. 




All of these fadtors relating, to challenge ^n be effected by management. 
It is apparent that initiation of challenge fequires mafia gelnent ' s strong 
participation as well as action to provide? freedom \nd tools. .These, in turn, . 
will permit the staff to respond effectively. A * , 

\ i, . . ' , * 

The > response to challenge will come from individuals or groups. This is 
what management is st^iving^far, and tt^-must j^ndle this portion of the cycle . 
wijth the , same diligence as was used in initiacxng the challenge. The individual 
.will expect the following from management: . • ^9 

1. A reception of his/her ideas regar<J.lesS of * past perform- 
ance or curretit position may be. ' - - 

> ' - .» 

2. ' An opportunity to try oiit -suggestions wit^lin the resources 
* available. 

3. An open mind to unorthodox ideas.* 

4. ^ Consideration of his/her- ideas regardless of.^differences ^ ^ 

in personalities. * • ' 

5. H^lp and guidance in obtaining services, equipment, or 
^ talent that ^ill enhance the suggestions; 

' 6.C FreedT5m^ ^respond, in keeping with his/her individuality. ^, 

^ Freedom to respond to a challenge' prollf'ably deserves ^special analysis and 
discussion in today *s environment. Consider the following: 



Over a 25-year period the average weight of the resear^lT^ 
proposal documents submitted to the federal government 
has increased by; a factor of 10 to meet bureaucratic 
regulations. 



Confusion in regu^lations or policiesr carries over into 
many actions -today, rather than permitting conjF^dent 
action . ' * ' ^ 



3. The **system" does not always reward ^thoge who respond to / 
' " a challenge with positive action, but rather rewards thos^ / 
who make "no ,mi'stakes.** " / 

' . * • / ! 

Reward is the most difficult pdrt of the challenge-response-reward cycle fpr^ 
. management, to^ implement. A psychologist recently said, "If we just kneW what \ 
kind of reward each new employee -really wanted and we could provide it , we would 
double the creative performance of out staff." The fact is: different -people 
want different things, njoney being perhaps*. the most common denominator since it 
oan be converted into Aeans for satisfying^a variety of specific desires, '^ome 
\ individuals, for example, want only a certain environment. Enthusiasm is gener- 
ated wljen the individual k^ows that the results of his/her efftorts (innovation, = 
' new contepts, creativity) are going* to b§ implemented. However, consider the 
common complaint of scient;,ists and engineers working in the laboratorj^ 
•boss is not interested in science*" Whether -the bosft is or is not really inter- 
ested i« a go6d question-. Unfortunately, if this ds indeed so, the bbss has 
inhibited creativity. ' . ' ^ . 
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Reward is the prime motivation of people, and it must be implemented fairly 
and consistently. Yet the speclTfic nature of the reward given to different peo- 
ple must be different; management ipust be flexible. Thus, it follows that: 

1. Response to a dhallenge must be recognized and rewarded 

management. * . . -* 

' . ' ' '\ 

2. Reward must be real and in* keeping with the challenge and * 
the response • It is easy to succumb to •rewar4ing "old Joe"' 
for faithful service while outstanding perfymaneg by a 

younger «*per son is' disregarded. ^ * . • * 

3. The nature of the Vewar4s available should not reduce 
technical or scientific creativity. For example, man- 
agement^' s demonstration that administrative positions axe 
more desirable than technical ones may stifle technical^ 

•creativity. , . • ' ' . * 

4. A variety of reward^ must be avai,lai)le; publicity, finan- 
cial reward, position, authd'rity, recognitipn, stature, 

and working freedom. , ' ^ 

The phasic needs for , an atmosphere that will maximize the creative talents 
of individuals or groups* can be seen in the characteristics of the creative 
individual. Interestingly, these characteristics align themselves generally 
with the categories of challenge,^ response, and reward! 

Is intellectually curious ^ 
Enjoys a changing -spectrum. of activities 

Defines problems . ' ' } Challenge 

Is strongly motivated 

Puts data together in different ways 

Is goal oriented * 1 u 

Is not afraid of new idea? .x j esponse 

§eeks recognition \ „ / I - ^ * 

I Reward * 
Wants to see his/her contributions implemented) ^ 

Management s role is to establish the proper environment or climate. To 
be successful, management should use the factors discussed above to orient its 
policies, decisions, and actions. These ^policies and ^ctions must stimulate and 
not restrain creative expression. And, indeed, strong action is required to ini- 
tiate and luain^ain freedom of action and expression. - ' 

An example of a major dislocation to the challenge-response-reward cycle 
is the following. A well-known government laboratory has a mission to develop 
through r & d increased practical use of a basic commodity — this is the cha(ll«nge 
Advancement and othef rewards^ for the individual are based solely on the number 
of basic research papers published by the individual-. However, while outstanding 
basic research is dolie by^'chis laboratory, the contribution of such research to 
the achievement of the laboratory's mission is nominal. . 
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H6w. Management Can Meet These Needs . - . . * 

Management's problem is to direct the operation of the r & d effort^so a§ 
to optimize the creative benefits of the chaj-l^nge-response-reward cyclg. This 
requires: 

* * I* 
A real desire 

2. Real reward mechanisms : * 

3. Creation and maintenance of a stimulating environment 

Desire on the part of mana'^ement to maximize the creative potential of a 
•company's staff is not to be dismissed lightly. One might say, "Why certainly 
. management wants to improve the creativepess' of the staff," but doing so is not 
easy. It involves. making difficult decisions, as well as an ability to resist^ 
pressures of a social, political, aud personal nature. Rationalization of suc- 
cess or failure of individuals must be minimized. Does this 'Sound' like a simple 
task? Add to this ope fact: that in general, no leaders or organization tak^ 
' energetic action to promote increased creati}firty/productivity . Even so, this n 
is probably the most critical step management must take. Its dedication to this 
goal will manifest itself to the staff both overtly and covertly«-and both kinds 
of messages are picked up by the staff. Answers to a few questions illustrate 
how management communicates its desire to enhance creativity: 

1. Does management visit the laboratories? 

' • 2. Does management seek the opinions of scientific personnel? 

?. What proportion of a manager's time is devoted to talking 
^to technical people? « 

^. What ate the relative cbnditions of the laboratory working 

areas compared to management's? # . , ' 

r 

Reward N in one form or another. Is the action taken by management to complete 
the challenge-responserreward cycle. Because people differ, seek different things, 
and contribute to tKe organizatibn 'in different ways, reward must take different 
forms. The first st^p iri, creating a reward process that will maximize creativity/ 
productivity is to build it into the basis for -advancement . Thus, the individual 
will immediately recognize that this is Hesired by management and will do some- 
^thing about it. The basis for advancement should be simple, clear, and consis- 
tent in its use. A suggested basis would be: performance on assigned i tasks; and 
.performance in* suggesting and/or initiating new scientific *and technical concepts 
and programs or operational activities aimed at achieving the organization's 

mission. ^ » ^ 

> • . ' ' 

Jhis basis for evaluating and advancing ^eople^is one of the keys to what 
management can do to ^maximize the potential for performing outstanding r & d. 
It is as applicable to a clerk as to a s6lid-sj:ate> physicist . ,It clearly indicates 
that management expects and ijR looking for initiative, creativity, and product iv- . 
Ity from the research staff; 

This suggested basis for" advancement within the organization has two addi- 
tional benefits: First, Its simplicity should minimize distortion of its message 
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by the ^oiranunicat^ipn network in the organization ysecond, fts expression of a 
desire fior creativity/productivity signals reward for those individuals ^hb are 

creativeV . . ^ • ' \ . ' 

\ ' - . 

To ptoperly implement this' Basis fof advancement is not £asy, Jt involves ^ 
evaluating the response of individuals and rewarding individuals with different 
desires, ^he most appropriate solution to the latter problem is to^reward ^.^ 
people on the basis ot their;^unctional accomplishments within the' organis^ation, 
this prob^lily being the area where they wish recdgnitiojri or at least where" the 
reward will* make sense to the indivi<iual, to his/her colleagues, ajfCS to'^tKe 
staff generally. Every organized gi^oup of researchers consists of -those people 
who provide services, those who perform administrative or loandgement;' functilons^ 
those who are concerned with marketing and sales, as well as those Vho perform^ 
purely ^scienif if ic and eifl^ineering tasks. Cannot an individual who demonstrates 
creativity, initiative, and accomplishment in any of those aijeas be revferded? 
It must be real reward for real accomplishment — no rationalization abdjut "good 
fold Joe." The seriousness with which nianagement implements reward can be indi-^* 
cated by the^^nswers to the following:^ * , ' • 

1. Are there real positions in the -olrgdnization for individual? 
•who demonstrate scientific accomplisTiment? 



2. 



Are t 
are c 
occu; 
with? 



le "scientific positions" occupied by individuals who 
eative and have initiated new concepts, or are they 
pied by persons whom management didn't know what to do 



4. 



Is the reward for outstanding scientific performance the 
advancement to an administrative job? ^ 

Do people wjio perform different functions, but n^aKe equally 
important contributiorts, have tomparable positions? 



To illustrate* how such an overt arrangement that presumably satisfies «^this 
type of reward can falter, let us consider the problem of giving proper recogni- » 
tion to the creative scientist. Which of the following statements indicates that 
management truly recognizes the creative scientist? 

1. "The position of research scientist is one in which we place; 
our sfeniorHPen who are not interested in administrative and 
management duties.^" \ " ^ j 

2. "Individuals occupying the position of research scientist are 
^ our>nost creative jQndivid.uaJ.s and are the source* of the new 

concjipts on which we, work." 

Creation of a stimulating "^environment is one of the' key objectives that 
must be realised if management des^ires to maximize .creativity . It requires a 
^continual innerplay between <^hallenge and ^^e^sponse, plus the establishment ol: 
effective sareguards against the deadening impact of confusion,^ misunders tam- 
ing, lack of decisions, and pver-administration. To enhance freedom of response, 
to challenge '/takes strong leadership by management. . Management must watch tlie 
organizational structyre and operation to ensure that it does not restrain tqe 
idividual, \ut rather helps and reinforces him/her. . * , 
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^ Minimlz^ing conflistibn facil^ates the performance of effective ^ygrk b^L • 
individuals • ^ Each petson w^nts to know what* is Expected of him/her andi tfie^ 
relationships with selected ^ther people within the organization.^ Does this 
take strong management? Certainly it does require establishing: 

- 1.^ *A clear under sta(nding of ^he responsibilities and author- 
ities o? different" positions. * ' , 

2. Positions that have real bases^ both in operation arid as ' \ 
% an expressiqn of performance'.. , • . 

3. Positions for "scientific*' as well as management-type \ . ' 
*^ - individuals. . . 

* ^ I ' • . 

^ A. stimulating environment conducive to high-quality research activity is 
one in which each individual has freedom and f lexibilij:3iL,-«S-^ell as the tools' ^ 
fpt doing effective work. This is difficult for" management to\^mplement in ^ 
view of tremendous pressures that combine to promote Parkinson*s Law. Adherence"^ 
to a few genersfl rules, howeyer, can he^lp^ , ^ - *e>* ^ / 

1. The span or control of , any manager or group leader sh^jld 
1 be so broad that he/she. does not have the time to do the 

wojk of subordirfetes. • . *V 

2. The organizational structure must show the importance of ^ _ 
the creative rese^cher. ' ' < 

' 3. ^ Whenever possible, groups should be oriented T:o a task, pro- . ' 

duqt, or goal rather than to be a techpique or olsciplinevj, , . . 

. Whan a gtoup leader is res'ponsible for' ^ reasonable* number of individuals, 
he/she "is forced to manage and l^d rather than to do the work of the staff. 
Conversely, the J.ndividiial9 reporting to' the 'group leader are presented \«rth* 
the, maximum challenge of performing with a minimum of supervision. Thus, ^his,. ^ 
arrangement promotes self-reliance agd indi^dual initiative. Further, those 
individuals who are truly prbductive will "sfahd out 6f the crdwd" ancj, con- 
sequently, management will find ft easier to identify, favorably evaluate, and^ 
properly rfeward them. * - - i ^ 




Creative research involves the puttipg together of ide:as and cohfcepi^s 
stemming from different sources and disdipliwgp. If groups are oriented toward*>j * 
a gene^r^i task: ox 'goal and are made up of individuals selectively combined ber 
causi&- t;heir talents aire needed to accompy^ the task; we' optimize creative 
effort. Among, the other bjenefits, the effort required to penetrate orgariiza- ''A^ 
tional barriers in order to communicate can be^ used -for other, more productive 
.purposes.- •^^ ^ , , , * ' '^^ ^ 



To continually stimulate and challenge individuals, management should: 

1. Rotate individuals to different tasks" and environments,. 

within reason. * ^ ' ' 6 * 

: ^ ' ^ ..." .4 



2. "^oramunicate regularly and decisively with the staff.- 

3. Establish a variety oi tasks, problems, and goals, the 
degree- o£ definition of each varying with the' type of 
•individual assigned* 

4. Be continually receptive to new ideas. 

5. Plan for corporate and business growth. 

The general. objective of these actions is to prevent individuals from • 
becoming stagnant as well as to remove barriers to creative expression.. Most 
people faced with a new challenge will -rise to.it, thus enhgncing their own 
performance and simultaneously^ stimulating others. ' 

Concl usion 

. » -» ■ * 

A challeng^-response-reward cycle is a sound theme or basis for managing 
r & d so as to abhiev6 increased productivity and creativeness. Truly successful 
mana^ment of researchers should be* devoted to the task of increasing the crea- 
tive research potential of the staff by applying techniques that are consistent 
with this cycle. This will involve an apparent contradiction — between maximum 
freedom and strong leadership. ^Strong leadership will be needed to properly 
challenge and reward individuals. This may be obvious. What may not be so ob- 
vious is the need for management to exert strong leadership so" as to guard the 
individuals' freedom of response* and freedom of creative activity. Without this 
response, management is not realizing the potential of the individuals and its 
effect on the viability of the organization. 
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Introduction 

It is probably no coincidence that the questioning of the effectiveness of 
educational research and deve^lopment expenditure which necessitates this piLper, 
coincides with a questioning of the effectiveness of the educational enterprise- 
itself. When times get rough the call goes out to begin studying those who are 
(doing the studying. The implication is that this latter group — educational 
r & d workers — ^were supposed to, but have not actually been producing solutions 
to problems in the educational sector in sufficient quantity or quality to fore- 
stall yet another "crisis in the classroom." 

Seven years ago Charles Silberman had-Jto in^ke a case, that there was indeed 
a crisis in the classroom.! Today, fewer Americans seem to- demarfd that they be 
convinced of this, indicating superficially that (a) things have gotten worse, 
(b) SiXberman was effective in making bis point, or (c) the responses inspired 
in part by critics such as Silberman have indeed bfecome part of the problem. 
In any case, the perceptions of the key problems wKich plague school systems 
have undergone a marked change in the pa^ seven years. Silberman, ani other 
pritics of the 1960s such ^s Jpnahthan Kozol, Edgar Friedenberg, and Paul Good- 
man, argued that with a very few exceptions, the Americail educational system 
stifled spontaneity and creativity, and thus made a mockery of what education 
was suppored to be about. ^ The titl^ of Kozol 's book-about the education of 
blacks in Bbston, Death at an Early Age , captures su<;:finctly the middle-class 
reformer's sentiments about what schools were doing to children, and not solely 
children in' inner city schopls. Melvin Kohn's research suggests, on the other 
hand, that not everyone viewed this as^ a problem*: working class parents in 
particular understood, such "stifling" (reinterpreted by Kohn as the inculcation 
of behavioral patterns of obedience, tieatness, and Conformity to an external 
autho^ity^ as part, and' parciel of what schools were supposed to do.^ In their 
view, school, like life, was no-t something to be enjoyed, but rather something 
to be gotten through. 

i * ' * ^ I - 

Influential business groups, however, were also dissatisfied with Americaij' 
education, not so much because it did not reward s^fontaneity , but rather because 
it Was getting too expensive. The^. solution which' they advocated was a federally 
funded e'ducational r & d effort which could produce and disseminate ne^jf^.tech- 
r><51ogies capable of controlling costs. The^ Committee for Economic Development's 
1968 Report, Innovations tn ^Education; New Directions for the American School ,^ 
is a revealing document. A major fear expressed in this report was that incre- 
mental funds to the. educational sector would be dissipated in salary increases 



and increased teacher employment necessitated by ineffectual attempts to 
improve performance by limiting class size* Rather, argued €ED, the key tp 
«^mproving education lay in improving. the productivity of the individual teacher, 
by increasing thfe amount of capital with whiclj he/§he coc^erated. The r & d 
effort undertaken in the educational sector was compared unfavorably with that 
undertaken by indust^ry both in terms of the share of total expe^rlditure (gross 
receipts) devoted to r & d and in terms of education's weak commitment to 
development as opposed to research* "The missing link in education is develop- 
ment research as it is practiced in industry," argued the report, pointing out 
that industry spent from 3 A to 5 percent of gross revenues on r & d versus ^ 
less than 1 percent in education* * Moreover 77 percent of that total in in- 
dustry was spent on development, as compared with only 10 to 12 percent in" . 
education*^ ^ 

The technocratic vision that r- & d effort could ^be jnobilized to solve ed- 
ucational problems in much the same way that manpower waS- mobilized to 'put a 
man on the moon lay behind the earlier establishment* of a national network of 
regional r & d centers (196*3^ and labs (1965-) , under the provisions of the 
Codjjerative Research Act* Business groups found common ground with those in- 
terested in applying the techniques of aeronautical or industrial engineering 
to educational systems, as well as with middle-clasps reformers interested in 
making the learning experience more Spontaneous and joyful* All were in favor 
of changing traditional methods of instruction, and could agree on the desir- 
ability, if not the content, of that catchword, innovation* 

Given this coalition for change, one might have expected dramatic results 
by 1977* The actual achievements have^been less Spectacular* Hardware advo- 
cates have had some success in developing and implementing computer^ assisted 
instruction (CAI) * But the urgency ancT feasibility of' such implementation is 
greatly reduced now, partly because the general economic downturn has made 
funds at both the federal and local levels more difficult to obtain, partly ^ 
because of a general reaction against "excessive"" innovation, and^ partly be- 
cause the end of the baby boom removed some of the pressure on the costs side: 
average annual elementary and* secondary 'school teachers' salaries, corrected ^ 
for inflation, peaked In 1972 and have actually declined since then*^. 

The reformers did win some of their battles for a more flexible,^ child- 
oriented education* Indeed, if there is 'one positive thing which can be said^ 
about American education today it is that students seem to be enjoying school 
more* One of the few current bright spots' in U*S. education is the continued , 
increase in school participation rates* Both enrollment and average attendance 
as a percentage of the population 'ages 5 to 17 have' continued to increase during 
the 1970s *^ One might appeal to high teenage^ unemployment* as the explanation 
of thiSrtrepd, but the number of high scfiool graduates as a p^^ent of all 17- 
year-oids has declined slightly between 1969 and 1975*^ Many high school stu- 
dents now hold part-time jobs, which was more difficult in the days of heav^ 
homework assignments, and the labor force participation rates among IG-year- - 
olds through 19-year-olds^ have actually increased in ,tiie past 5 years *5 It i3 
then at least possible that school has been made a^-more. pleasant , if lesa de- 
manding environment* On the other hand, the increasing rate at which young 
people take their own lives cautions against facile generalizations about trends 
in the happiness of the -schools age population* 



Suicide remains the third leading cause of death for males 15 to 24 years of 
|ge, ranking after accidents and homictde. And that rate has been increasing, 
from 13.5 deaths per 100,000 in 1970 to 15.7 in 1972, to approximately 17.2 
•in 1973, •'•^ the latest date for which detailed statistics «re available. 

Whether students are^appier now* than in previous years, parents, tea^ihers 
and property-tax payers do not seem^to be. This is manifested in widespread 
concern with declining test scores (not all of which by any' means can be at- 
tributed, to the inceased numbers of students taking these tests),, violence irr 
the classroom, grade inflation, and a generally perceived decline In "stan- 
dards," which many associate, rightly or wrongly, with the reforms inspired 
by the critics of the 1960s. By 19Z6, according to the Annual Gallup Poll' of v 
the Public's Attitudes Toward Education, 59 percent of the American publicf 
felt 'that declining national test scores indicajfeled a decline in the quality 
of American education. In both 1970 and 197^/ according to these same polls,' 
the American public most frequently identified lack of discipline when asked 
enumerate the major problems facing American public schools. Indeed, th% 
share of respondents mentioning this problem rose from 18 to 22 percent* over 
this period. But even more dramatic was the increase over the same 6-year 
span from 6 to 14 percent in the share of respondents citing poor Curricula . 
as a major problem. Integration and insufficient funding were each mentioned 
by 17 percent of respondents* in 1970, *but only by. 15 and 14 percent^ respective- 
ly in 1976.^^ ''The educational system has shared, along with government, 
business, medicine,^ and the press in a general .decline in public confidence 
in major institutions, p£lrticularly manifested in the years 1973 to 1976, 'dur- 
ing, whlcii period the share of respondents ^ha^ing "hardly any" confidence in 
* those running educatiofVal systems increased from 8.2 to 15.3 percent. 

Whatever the causes of these developments, schools have, in the p^st dec- * 
ade^been faced with taxpayer revolts, in addition to^ tHe turmoil resulting 
from catcher* stril^es , desegregation crises, and^a host of otfter problems. \; 
Whether tfi^^Jevelopments add up to a crisis, and whether there has ever/oeen 
a period in^&^Jxan education .which some did not view as a crisis situation, 
are questions that are not easily answered: let us agree that American educa- 
tion, faces today problems sotae of which are different or more acute than those 
experienced in earlier periods. One of the^^most serious problems from the 
s£andpointi of educational r & d is a reaction against two decades of innova- 
tion, a'general' fatigue resulting from constant, and often contfadict9ry , 
change. , , ' . t 

It is absolutely essejit^ial that the effectiveness of educational r & d 
be discussed against the backdrop of the problems facing the educational sys-. 
tem itseljf. We can all'agree, very' abstractly , that the^ purpose of educa- 
tional r & d expenditure is to "improve" the educational process. But any 
attempt to measure the productivity of this effort requir^ the definition of 
outputs, or improvements. This cannot be done without *a clear view of ^he 
objectives of the educational .system and the problems faced by teachers and 
students in achieving these. - V 

• In a later section of this paper I will address the conceptual and tech- 
nical issues associated with the evaluation of the effectiveness of v educational 
r & d expenditure. These include the definition of the outputs of 'the educa- ^ 
tional system and thus, derivatively,-, of th6 educational r & d effort, and the 



related concepts of efficiency, productivity, and benefit/cost analysis. I 
will also disicuss some specif ic* ways in which one might icqxrove productivity 
by cutting costs and/or increasing^ benefits. \^ 

Before beginning to discuss Chese issues, however, it is worth asking 
whether, .assuming we can do, it, this' effort is a'nec'essary or desirable one*. 
Some economists would shudder at the thoughit-* that such a question should even ^ 
be asked,-* ^onomists t%^d to argue that efficiency is a good thing", and 
whatever can be done to improve it needs no further justification. Yet, if 
for 'no other reason than that we are not all econoniists,« I think this issue 
isn^orth presenting* 

It is naturally* somewhat distasteful for those who form part of the 
educational r & .d establishment to heed calls that the effectiveness of their 
labp-rs be evaluated. And yet, even participants in such efforts, surveying . 
on the one* hand, the variety ^nd extent of educational r & d expenditures in 
the last- two decadds and, on the other hand, the disarray in the. schools them- 
selves — £he declii;iing test scores, the violence, the rejected bond issues, the 
apparent drop in public confidence in school systems — cannot help entertaining, 
if only for brief moments, the null hypothesis that this whole r & d effort 
(if not the educational enterprise itself) has been a failure, if not a gigan- 
tic rip-off. And if it's a rip-off, th^ talking about productivity is a waste 
of time, since Output cannot be i4entified, and the maiii function of the r & d 
effort being that it keeps r & d workers off the streets. The more money spent 
and ^ the more .people employed, therefore the better. 

Even in less cynical moments, many can think of more important concerns 
than the productivity of expenditures on educational r & d. One could even - 
appeal to the macroeconomic lessons of* Keynesian economics as justification " 
for a less than wild enthusiasm for efficiency measures. Why should we worry 
about mitior improvein6nts in the allocation ofvjexisting employed productive 
resourc^, when any possible efficiency gains are overshadowed by the 'economic _ 
loss represented by unemplojted laboi; and unutilized capacity? A complementary 
full-employment variant of^ this argument compared any potential efficiency 
gains with the cost of one Trident, suKmar^ne ajid suggests that those wishing 
to im|Jrove the efficiency of government expenditure should pay more -attention 
to the Pentagon's budget, and getyoff t;he back of the NIE. 

4 \ • ' 

These 'last two arguments assume that 'educational r & d expenditures can- 
and have ihrfSe a positive contribution to an as yet undefined o.ut^ut. On^'the , 
other hand," there are those such as Milton Friedman who argue,- not all that 
facetiously, that government is the root of all evil. To the extent then, that 
education" and educational r & d are government coritrolled or influehce^, at-r- 
tempts to improve productivity and effectiveness of educational r & d/are worse 
than a waste of time, since government 'can only produce bads, and the more 
efficient production qf bads represents a net social* loss. ^ ^ 

*) 

If the purpose of educatioaal r & d is to "improve" the educational process 
we can thus distinguish between those who believe that r & 'd work Ca). haa had < 
no discelrnible effect on ^he process, (b)* has had- a positive impact," and (c) 
has, on balance, been detrimental to the process. My sympathies are with those 
who adopt the second position as, I suspect, are those of most participants in - 
this conference. Friedman^s argument,' which reaches position (c) on a priori 



grounds -strikes me as nihilistic. Governance, like the poor, you shall always 
have yo.u; the public policy debate ^o^ld center aroufid- its formsV not with 
its desirability as compared with a wc^irld of ma^rkets, presumed capable 6f 
functioning in its absence.. Some, o£ course, reach position (c) on empirical 
grounds, linking current problems to the reforms which came out of the sixties, 

. and those reforms to an earlier round of r & d. Whether ^this line of argument 
Is legitimate' or not, the very real problems which school systems face foday 
are not evidence against the proposition that things might have been worse in 

" the absence of reforms and in (the absence of the continu4hg r & d effort. Prog-* 
ress in some areas has been made, perhaps,^ however, at the expense of progress 

' in other areas. . • * * 

Nevertheless, ^we should retain a certain skepticism about the underlying 
premise of this discussion: namely, that current educational problems result 
from ►a failure of educational research and development, in turn attributable to 
a^misallocation of resources among various ^ projects, or lack of a sufficiently 
higti* level of overall funding. Participants in educational res^rch and develop- 
'ment are in something of a bijid. If, in obtaining funding,, more is claimed for 
research efforts than can possibly be delivered, then members of the r & d 
establishment risk being held re^onsible for tbe' tills of the scJhool system 
either because they failed to anticipate and provide solutions to problems ad- 
mittedly not of their own. creatjion, or because ne\< 
successful implementation of prior r & d effort^, 
on the other hand then the rationale for and prol 

may become less. * Giv^n this bond^ and given existing political and economic 
realities, some self-examinatiqn of the oMjtctivei^tid methods of educational.* 
research and de.velop'nent is necessary. Such sc^runiny, dis*tastef uj though it may 
be, is one of the better me^ns of preventing th^ receipt of one of Senator 
Proxmire's less than coveted Golden Fleecte awards^^^ 



problems are attributable to 
If claims are^more modest, ^ 
ability of public funding 



^e. Goals of an Educational^System * 

Accepting the position that educational> r & d e'xpendlture has made some 
contribution to the conduct of the^educational process, we return to the que8- 
tion of whether attempts to imprdve the productivity and efficiency of such* 
expenditures are desirable. The answer to those who point to unemployed re- 
sources and/or Trident submarines is that^ political and economic realities are 
such "t^at the issue simply mu§t be address'ed if a continued flow of funds is to 
be obtained. But who should do the evaluating? The debate developing around 
educational r & d evaluation replicates one which h^s chara'cterized the analysis 
of .r & d efforts in other areas. Are thpse conducting the research to aj^locate 
resources and judge ex post facto the ef fectivieness of s^uch allocation? Or is 
this to be done by others, outside^f , or peripheral to the tnajoi; research and 
development, efforts (economists for example)? Those directljr* involved in^the 
research effort also have the most specific knowledge of research problems and 
prospects,, as well as'possible conflicts of interest. 'Economists may be more 
disinterested, since the^ conduct a smaller proportion of educational ^r & d, but 
one should not assume that they have all, or-^ indeed any of the answers. They 
have, moreover, repeatedly demonstrated that excessive faith in the utility of x 
their discipline for technology assessment is no substitute for knowing-'what they 
are talking about. ' Nevertheless, economists do have a number of concepts which 
may be useful in organizing this discus.slon. 
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Prolijjctivityi efficiency, and benefit/cost analysis are three of the most 
important of ^ these. In its simplest .form, productivity is admeasure of output 
per unit of input. It is* most easily thought of in physical tetms. A technique 

* or organization which achieves higher output given certain inputs, or achieves' 
similar outputs using fewer inputs, 'is more efficient in a technological sensje. 
Benefit/cost analysis represents an attempt- by 'policy atia-Wsts to apply these 
concepts to the actual problems of evaluating prospective policy decisions or 
programs of funding. Benefit/cost calculations require the' Identification of 
output (benefits) and inputs (costs)*. A program-with higher benefit/cost ratio 
represents a more efficient allocation of resources, . \ 

In the cas^e of educational research an4 development expenditu^re , any at- ^ 
-tempt to evaluate productivity and effectiveness requires an identif-ipation. of 

* the outputs (benefits), associ^ted'with such^ expenditure . The UlE Databook 
(1976) defines tlje goals of educational & ^ T^^^®^ vaguely as the attempt to 

* "unde^rstand, influence and produce educational improvement."^^ "Educational 
improvement" is ^ surprisingly vague. -term with which to begin a document which* 

, otherwise abounds in^ sp^ecifics. But , it does remind us *again that the effective- 
*hess of educational r 6i d expendituffe has to^be evaluated in terms- of its im- 
pact on tfie educational system itself. ,The demand for educational r & d is 
4erived from the demand for education. It is impprtant to come-back again and 
again 'to this point, lest we be, mesmerized by process and^ lose sight of objec- » 
tives. We must begin with a clear ^notion of what the school system is supposed 
to do. . ' . • ^ ^ ^ 



Let me suggest that it is not Jthat difficult to define.. the goals of our 
educational system. Our system should provide pupils with'the Sets of basic 
skill's and analytical abilities, necessary to fulfill" job requirements and the 
-requirements of citizenship in a d^nrocratic society ^ Primary schools (and 
increasingly preschools — vide Headstart) should provide all students with a 
platform upon which all those williijg and able* to go on to advanced training 
cay build. 'And schools should be^conducted in an orderly fashion, not simply * 
• becau^^the previous goal,s can be^ fulfilled only with difficulty if the insti- 
tutional setting Is chaotic, but. also because schools are an important social- 
izing ageiicy themselves, and children must develop to some extent the internal 
restraints'" necessary to function in a well ordered classroom* if they are tp 
make a productive contribution* to the economy and society. I do hot mean to 
imply that open classrooms cannot or should not be* implemented or that we should 
discourage spontaneity and creativity. But even critics of the American educa- 
^ tional system recognij^e today in a way tfiey-often did 'not in the 196ps that 
schools are intrinsically constraining. .Compare for example, the title of. 
George B. Leonard's^ 1969 book. Education and Ecstasy ^ with Samuel Bowles and ,g 
Herbert Gintis! 1976 comments on this poj^pt 'in Schooling in Capitalist America . 

EducatiotxaL philosophers and ^iti^ens' in general may and should debate the 
relative Imgortance of thes'e* various functions and objectives, as well as such 
centVal questions as the extent to which schools can or should be tMlored or 
differentiated to siiit children of differeni: backgrounds, and also the extent 
to which schools .can be effective mechanisms of change themselves. Some con- 
sensus on tljese goals', however, is essential if the productivity and effective- 

, ness of education, and of educational r & d is tO' be treated' as a technical 'and 

.» * ' • 
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not a political problem.^ Even then, diff iculj^^udgraents may have to be ren- 
dered where improvement* in o^e objective can be obtained only at the expense 
of another. . ' ' ^ ' , 

X5iven these definitions, we can decompose efforts to improve productivity 
into those which reduce costs, given program commitifients, and those which in- 
rease benefits, given expenditure levels." In practice the overall responsi- 
bilities for improving benefits and cutting costs should perhaps lie witl) dif- 
ferent groups JLn the funding r & d organizational structure. The individual 
researcher may have a vested interest in maximizing the value of his/her grarit 
or contract, but also probably knows where costs can be cut with t:he least 
damage. In experimental work, smaller samples cap[^^>e. drawn, fewer interviews 
conducted, or less complex calculations made. The researcher is probably most 
aware of .how to effect such economies with mijiimal sacrifice to the quality of 
the' final research output. On the development side, the military and space 
efforts remind us of the important trade-offs between the rapidity wlxh which 
a product is develop^ and the expense. We may b^ able to obtain q/ick results, 
but only at very high cost. Less ambitious timetables, then, ar,e one way in 
which costs can be trimmed.. Once again, it is the developer who is probably 
best informed about where economies can 'be made with the least efjffect on the 
quality of the final product and the datie at which it is implemented. Admit- 
tedly researchers and developers may be reluctant to reveal this inf ormatipn 
and apply it; it is hoped they may be subject to persuasion. 

It is perhaps 'asking too- much, however, of researchers (and to a lesser 
e^jtent developefs) that they compare the value of their proposed work with 
what might be obtained making alternate use of public funds. Researchers' have ^ 
a 'tendency to belieVe that any' project on which they are willing *to commit -their 
time and effort is necessarily meritorious, and it may well be, in terms of 
pure scientific value. It is really the responsibility of the national funding 
agencies to strike a precarious balance between « the support of mission-oriented, 
or applied research, and more basic research. Such choices involve decision 
making under uncertainty, and should be made using the best scientific, en- 
gineering, "and organizational advicS available. But it is not a responsibility'' 
which can be abdicated. Funders must take a hard and comprehensive look at 
the benefit side. Much academic research can be .justified on the basis of 
scientific mer^. * This does not necessarily warrant federal support. One of 
the most salient arguments for establishing a National Institute o^ Education 
was the need for a clearly atticiilated and coordinated r & d policy*^^* Tt^ 
is not clear that thi^ has yet been achieved; thejg^„j§^d effort can still be ^ 
criticized as haphazard and piecemeaX. * ' ' \ 

This not to say that definition dr^r & d goals should be sought for the 
6ake of definition. A (controlled eclecticism may be in order. Many in the 
educational r & d establiahfaent seem particularly enamored of the military- 
space- industry model of the r & d process and, furthermore, seem to have become 
more so in the past several years.' ? Compare for example the 197.0 Off icq of Educa 
tlon report on Educational Research and Development in t^e United States with 
the 1976 National Institute of Education Databook; The Status of Education Re- 
search and Deve^lopment in the Ugited States . The latter document speaks almost 
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exclusively of r & d outputs as "products" to be sold to users. To complete 
the analogy, research seems to consist ofi^ developing Qew products; development 
with market research and sale. . The military-technological-industrial model 
appears to have a powerful hold' on those curtentJy thinking about educational 
r & d. The earli-er document evidenced a more catholic interpretation of what 
educational r & d consisted of.-^" 

_ , It is worth remembering .that tjae modern conception of how innovations are 
produced £s comparatively recent. ^ Most innovations in the eighteenth and nine- 
teenth centuries were due to the efforts of creative, nonformally trained in-S'f' 
dividd^s tinkering on a part-time basis, often alone. We must at least 
recogniaje the possibility, as does Jlelson, that "the understanding and tech- 
nique wj|ll evolve. . .through the^ experience and sharpened imagination of people 
who are /Actually trying to deal with the problems, not through the data ga*ther- 
ing and theorizing of people with fancy degrees doing something called r & d." 
Even in the twentieth century the r"& d structure on which the educational 
effort is now modelled has not always lead to successful in'novation. Massive 
r 6t d support for the ^itish ai^rframe industry, for example, led to a series 'of ' 
technological successes but commercial failures ,^1 Similarly, many military ^ 
•r 6t d projects have had to be abandoned, or have been* carried through only, at 
enormous expense', producing filial products which did not live to exp^tations. 

finally, W should be aware that even a heavily supported r & d effort may 
simply be unable tq produce. some of the innovations expected of it. The develop- 
ment of an effective teaching machine which could even approach teacher replace- 
ment ma^ be be^jond "existing or foreseeable engineering and scientific capa- 
bilities. Spme problems whicff appeared easily spluble using modern copiputer 
technology have turned out in practice to be much more intijactable than. had 'been 
anticipated. The translation of language by computer is a prime example. 

Extreme pessimism is not warranted; a good deal can and has been done. But 
it must be recognized that since educational r & d covers a wide rang^ of acti- 
vities, the organizational forms suitable for setae of these activities may not 
be suitable for others, what are some of these varied activities? We have at 
one ^extreme those who c one et;jithems elves basically with how dhildren learn, and 
at, the other extreme with what^ey should learn,.' Most "pure" research is done 
by those with advanced degrees in education or psychology and is^conducted with- 
in an aca.demic environment. ' Such efforts haV6 in the past yielded ^'extremely 
useful information, in particular, a greater appreciation of the importance of 
the preschool years jpti the pro<iess of child development. This rese^ch formed 
part of the rationale for such programs as Headstart. .At the other end of the 
r 6t d spectrum, we have those concerned with what children should ' learn.' Here 
again, there are some successful models — the PSSC, BSSC, and^ SMSG efforts in , . 
particular; It is obvioiisly not very useful to xio>a good job teaching children 
incorrect or obsolete information or analyses. TK^ most notable successes so 
far have been in the ftard sciences; such curriculum development may be more dif- 
ficult in the social sciences where there is less consensus on what constitutes 
received doctrine, although recent textbook controversies indicate that even the 
hard sciences are not immune to Such controversy. It may also be more difficult 
to enlist competent personnel for curriculum revisions below the high school 
level. " ' 
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In between these two poles are thSse wKb concern themselves with the. best 
means of* transmitting curriculum to childi^n:' t\iey develop and study the ef- 
fectiveAes*s of such^^nnovations as different forms of* classroom organization, 
language labs, computer assisted instruction^ ,or the use of publi^c television 
(productions of the CJiiWren's Television Workshop being a successful sample) • • 
Finally; of course, some educational research consists primarily of data 
gatliering: prt>ducing an information base which may^ faciliate some of the r & d 
efforts just described* ^' , ^ 

Th§ diversity of this effort should be respected. It is somewhat disturb- 
ing that the most'well known and sy.^tce;;5sful of educational innovations in the 
past two decades appear to have orig'inated in the late 1950s and early 1960g^ 
»as the result .of research authorizations under the Cooperative Research Act 
of 1954 J (as amended), also the period in which the National Science Foundation 
•gave its heaviest support to curriculum development in the sciences at the high 
school level. Most of this work was done through contract or grant arrangements 
with colleges, universities, cristate educational agencies, ritlier than a net- 
work of'r & d labs and centers.* Some of the innovations to. come put pf the ^ 
r & d labs and cepters in recent ^ears may evet^tually prove t;hemselvds, but the 
fact that their Introduction has coincided with an increased^ubl^c concern with 
inadequate curricula is at I'east a cautionary mfce.^^ \ 

Witfiin the terminological framework of benefit/cost analysis, the responsi- 
bility of improving benefits given expenditure levels belongs primarily with the 
jj^tional* fiund^}ng agencies, partly because they control the preponderant share 
,ot resources going into educational r & d and partly .because they, as opposed 
'to the rert^aroher or developer,' can expected to have an OverView of the . 
process. This responsibility entails decisioijj^about organizational structure, 
dir^ction^ ^f\s^& d,. and Implementation. Cost cutting^per se may keep us within 
liwered-budgetar]^ commitment, J>ut will do little to improve the overall research 
effort.^ The deci6ioi\3 are not easy:/ do we wish to concentrate on the develop- 
ment of ijtew^toducts — curricula or blueprints, or should we rather build from 
^ stifeiigtlH^t:0nfeentrating ii|^>th^B implementation of proven programs, those tested 
andlfbund-^^ucces^ful g^lready? 4r ' ^ 

« - . • 15 - 

^ In^dybajrioi^ as well as elsewhere, th0 problem is in part one of leader- 

jsnip-^-n^ ,^!Uaders^iV. f or the - sake of leadership, for it is better to wander aim- 
lessly 'than -march in unison 'down blind alleys, but effective visionary leader- 
s^£pS:*"^ecl8i<m makers with the tfapability of transcending the matrices of 
^individual idtel^est, seeing to the heart of the problems and identifying the 
C^reas-whiG^ are /most lively to rapay a wise investment. Implementation, like- 
•/ise, requires- leadership. 'School systems are organizations, and they have 
limiteicl ^a^abiliti-es for absorbing innovation. The funding agencies, in addi-'isii^ 
tion to ^^ilocating research funds*, have a responsibility much like that of the 
FDA to- scffeen oiit ' potentially harmful new product^. The danger in the educa- 
tional sphere is less that ill-conceived innovation will permanently damage 
students-^'ir% would, their it^gestfon of a toxic substance, but rather that un- ' 
necessary change^ saps the abilities of orgaftiization^' to absorb future innova- 
tions w^ich are potentially far more beneficial. ^ ^ 



One/sho|(i^ use^market metaphors with care in discussing educational^ 
r & d. Selling ^ reorganization or^new^product to a, school system is notL the 
• same as selling a marginally . useful product to the. American consumer. ' The lat- 
ter does not seem adversely ta--^fect the consumer's willingness to purchase 
future products. Most educational innovations, onrthe^other hand, even^f 
centered aroypd hardj/are, involve reorganization. Frequent and/or ill-advised 

. reorganization Increases future resistance to ^ch efforts, and reduces the 
possible benefits to be obtained. This resistance may take ^overt or covert 
forms, and is complicated by the fact, that ther^ is relatively little direct 

'control over what goes on in the classroom. Unless innovation is conducted' 
carefully and selectively, fut(ire» options may be foreclpsfed. 

When Ljmdon Johnson declared war on poverty, a great many people took 
him seriously . When Gerald- Ford urged us to whip inflation now, fewer listened. 
When President Carter declared his attack on ener.gy to be the moral equivalent ' 
of war, many people went 'to- sleep . And yet there Is , every reason to believe 
* that the energy problem is a potentially v*:y serious one. The situation with 
respect to educational innovatibn -is similar.' fn our political system, ef- 
fective innovation recf&ires commitment from a great many .different individuals, 
and the ability to organize that type of .commitment depends to some degree on 
how frequently it is demanded. This problem is compounded today by the diffi- 
culties of continuing ^innovation and reform in an essentially conservative era^ 
an era of retrenchment in which the ^ commitment to, move* forward in building a 
better society seems stalemated or in retreat whereve* we look. Money is, of 
course, imp*drtant in implementation, but it-^ is only part of the formula. y 

. In thinking about the milifca^ry or the industrial models of r &^d, note 
that the educational product development^ cycle diffei^ in important regards 
from either. Education shares with military-space- r & d the fact that govern- 
ment provides most of the r & d funds. It shares with the consumer ^product 
development cycle a large number of final consuming units, in contrast to the 
military-space model. It shares wfth neither a preponderance of not-for-profit 
organizations in the actual conduct of r & d. These difference serve as a 
reminder that the educational r & d effort is a unique and diverse one, and^ 
we would be ill'-advised to force all elements of £t into a prec^ceived mold^ 

DSrelopment without implementation is tragic if the innovation is a good 
one;dlts. a bl^fMihg if it is marginal.^ Innovations must be ut^ilized, however, 
if they are to shoy up in the benefit column of any benefit/cost calculation. 
The funding agencies, with the advice and consent of the r & d community, must 
^make critical decisions regarding the focus of researcK, development^,, and im- 
plemei^tation. But action for the sake of action may be worse than no actioi> at 
all. In France or the Soviet Union, the decision of one government^ mit(is try ' 
assures the adoption of a new text throughout the^ country 's school system.' The 
absence in the United States of a federally controlled educational system im- 
parts some bias agains't. new innovation which can delay the utilization of poten- 
tially beneficial techniques. It also provides some .(although it Is not clear 
how much) insurance ag^ainsm^atastrophic failures. 

Conclusion 

. In concluding, I am reminded of our re'^onsibilities to be clear, concise, 
and to the point. We wish to avoid the convening of conferences on the 



productivity and ef fectiven^j^of ev^uatidiis'of the productivity and effective 
ness of educational^research and development. In this spirit I will' summarize 
the main points of thi^ paper. First, the productivity and effectiveness of 
educational research and development is primarily a function of its impact on 
the ''educational system itself. In tKe short run, cost savings on existing proj 
ects may keep us within budgetary limitations. They /Jo not, however, necessar- 
ily provide the key to long run productivity ^improvements . Second^ what eve^j|^ 

"the unpleasantness which, results from cost cutting, and benefit evaluation, sue! 
steps ar^ politically necessary if funding is to be sustained ^and,_4ndeed, if 
the claim is to be justified. Third, the educational^ research and deA^elopment 
effort is a diverse one, and this diversity should be \ respected . No one* or- 
ganizational form is correct for all' elements of this V^^ocessl Fourth, ill- 
conceived innovation is worse than none at all. It is doubly damaging in that 
not only may it lead to performance loss* in the present, but it may also fore- 
close options in the future. Finally, to assure the long run direction of ener. 
gies toward areas which will yield major benefits, we need courageous, wise 
and, perhaps, also lucky decisions; the supply' function fdr vhich seems un-. . 
fortunately to be rather inelastic. Let me close with an appeal, once 'again, 
for *a controlled eclecticism, and ecleqticism which permits direction in our 

, efforts, yet flexibility enough when conditions change, to fllow for course 
alterations. ' ^ 
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r ' Session^ 4 

another view. on productivity and, effectiveness in 
educationXl R & D • 

LesliB 'Salmon-Cox ^ ^ 

j Learning Research and Development Center " . 
University of Pittsbuirgh 

In his paper, Dr. Field emphasized that product i^^ity and effectiveness 
must be seen as a function of the impact of educational r & d on the educa- 
tional system itself . I^ think thi^ is true; but it is not enough. In order 
to amplify that view, my remarks will mesh, in some places, and complement, 
in others, w^at he has said. When what is 'being discussed is educational ' 
r^& d, then I think it important to keep several different reference points' 
in mind. One ver^ important one is the short history of this particular 
knowledge production system itself. . ^ ' 

First, a brief word on my own orientation to the question. ^ Even £hose 
familiar with- the X^eanjin^g Research and Development Center (LRDC) — with w^iat 
^it has already produced and is currently pursuing — may not be familiar with 
th^* organizational meclfanism my wotk represents. » ^ 

Sinde^^it inception, LRDC has had a Board of Visitors', r noted ^scholars in 
the fields represented in the center. Ralph Tyler was chairman, of this bOdrd. 
for its first, 10 ^ears; Wayne Holtzman is the current chairman. The boarcfTias 
dad a resident staff Qf sociologists who "live" in* the center. What this means 
is. that the organization has been continuously studied for 13 years by people 
whose perspectives are the sociology of knowledge and of organizations. Our 
methods are anthropological; the center is our "tribe." • 

• ' . . . 

* o This has served to make us very familiar with knowledge production processes 
'in the*one case, I^C, and somewhat knowledgeable about* the 'field as a whole* 
The center's history having beei^ one *of the first 't:wo .centers funded Jji_1964, 
ispans the entire perioSt of federal, and other initiatives to vastly indredse 
the size and the impact of educational r & d. This is' the history of tfie Im- ^ * 
position of the ^'scientific modfel" on educationi 

Questions, regarding productivity and effectiveness must be asked within , 
the Context o£^ understanding of organisations and' goals.* tor educjational r & d, 
this^means some understanding of the field as a who^e, as well as of its com** 
ponent organizational parts — the '^4y these are structured, the' variety of mis- 
sions or objectives they comprise, the history each ericomp'Ssses . Dr. Fl^d 

: . . . ^ 

Preparation of these remarks waig supported by the Learning Research and Develop- 
men)K Cen^ei;, suppojrted in part as a research and development center by funds 
/rota^ethe' National Institute of Education (NIE) , United States Department' of ' 
^Health,, Educatibn,*j|^d Welfare* The ^opinions expressed do not necessarily 
reflect the posltill^or policy of NIE; and no official endorsement shouLd be 
"inferred. ' . , • • * < 



touched on this in his discussion pf the eclectic- nature of the r & d system. 
Taken together, the prfesentations at this conference reflect this diversity. . . 

* " 
— I shall discuss s5me evidence of productivity, largfely defined in terms 
of characteristics of knowledge production. In a young field, one looks for: 
the development ^of new paradigms, or 'the extension of . or ^borrowing from others , 
of ol^ ones; changes in standards of judgment; system growth and differentiation 

Looking quickly at, the field as a whole, we can' start witlythe reasons ' 
behind those early, and later, federal initiatives. It has been weil docu^ 
mented thaf prior to the Cooperative Resealrch Act of 1954, and betfween the 
passage of that and the initiation of the center's ^program in 1964,. educational 
'research was.-f ragmentary , noncumulative and, ^^increasingly had become out of 
touch with advances in the social/behavioral sciences. (This,' with «some rare 
excep'tiona> of course.) . * 

• 

The center's program was intended not just to produce specific innovations, 
but to impact the field, the field of educational res^drcK and the field of 
educational practice . The . idea was to provide: , fftrticture, leadership , and 
new models and standards of judgment. Field has alluded i:o the f^csd^t^o of , 
these in his paper, but not to the last, which J.s a significant one. <^ 

Starting with this last, standards of judgment, theie are some interesting 
indicators of the effectiveness of the,r"& d movement.||Leaders of laboratbries 
.and centers, and other institutionalized r & d have made prominent new stand- 
ards of quality in applying scientific concepts aad methods in education. For 
example, it is an unquestioned part of the r & d cycle that innovations — new 
curricula or other newly developed products — will be' pretested, systematically 
developed, »and evaluated, and that they will not be introduced simply on the 
strength of enthusiastic commitment. This emphasis on fairly specific concep- 
tions of standards sets the r &\d reform movement in education apart from* other 
efforts to improve American education as, for example, in* the 'tfree school" 
movement. 

" . The systeihatic 'development of curricula, of tests, o^ instructional modules 
of administrative innovations, has begun to have tracieable effects within in-- 
dustry, publlshing'^houses, and among sdhools ojE education, institutions training 
the next generation of teachers and scholars. ; * * ^ 
• f ' ' • * 

' A salient characteristic of thfe field in the past 15 years has been the 
growing prominency of people in the forefront of -the .traditional behavioral 
science discipliiies conducting work of relevance and importance to , educational 
problems.^ It la these traditional disciplines, e.g. ,. psyclio logy, that are 
the^custod^4|n9^f standards and criteria of e*xcellence in scholarship. .And, 
it is through the liicreasied presence of\ researchers from thesef fields that 
8cient;ific stidttdards have, become the standards for educational r & di 

Whatf:;.the r & i movement has made possible is a 'fostering^ of communication 
between the behavioral science disciplines and. education. It is, possible to 
se'fi now the changes which have taken place, when one ^compares the r & d centers 
currently, with their rep res,ent4|?ion -from ^psychology and sociology and a few 
other disciplines, with the rather parochial natute of 'educational r & d .at the 
time the centers were established^. ^ . , 



It is an indicator of the better position of educational research, as. 
an intellectual pursuit, that leading scholars in the behavioral/social 
sciences now address questions of reading instruction \or the measurement of 
school program effectiveness in ways not previously undertaken • In additfbn, 
there, is gBowing evidence of the influence of educational ;Concerns on the 
basic disciplines, oa the way questions are asked or research designs are 
iramed. • • ^ ' 

It is interesting also* that there are areas in which the traditional 
.experimental design may nowtbe seen by many as less than appropriate, e.g«, 
in many studied of school eff ectiveness, 'Yet, the need to counter the experi- 
mental paradigm^ and' the attempt^'to come up. with an equally rigorous if 'dif- 
ferent design has, it is generally agre,ed, improved 'the quality of this whole- 
area of research. * 

• There are other structural changes as well, ' In this vein changes in the. 
organization and functioning of the American Educational Research Association 
(AERA) — the professional organization which represents this field — can be 
taken as indicators* of the effectiveness of the move to bring science to bear 
on educational concerns. These changes reflect changes in knowledge production 
in the field in general. 

As the field of educational r & d has grown, so too has AERA — 'in its 
membership , its number of divisions, its annual meeting attendance. Without' 
enumerating these spe^cifics, ,the important thing aibout the association, -as 
about the field in '^general, has been Its growth and its differentiation. 

In the past decade, AERA has developed more and more specialized divi- 
sions, special interest groups,, and a larger and more diversified set of publi- 
cations. The literature' on modernization and institution building talks of 
the coming to maturity of societies and social groups arid points out as indi- 
cators of this process: system growth, systemic differentiation, and resulting 
institutional specialization. can view these also as criteria against which 
to measure growth in a developing* intellectual field such as educational r & d. 

The r & d system for education has gtrown tremendously ia 13 years and has, 
in fact, developed a strong leadership, in part reflected by the membership of 
this organization, CEDaR. It has developed 'a fairly differentiated structure, 
brgaT\izations of several kinds to meet varying needs: labs and centers, the 
ERIC system, field based programs such as Follow Through and^ maiiy more, ^nd 
if has more significantly set high standards and criteria of judgment regarding 
scholarship and production. -* ' ^ • 

Any analysis of the curi^nt state must mention the need -for: including • 
a greater^A^iety of the social sciences in the knowledge production process; 
working to bul.ld more extensive communication between the r & d system and. 
schpols across the country ;• working to better communicate the significance of 
work past and potential to a lairger segment of the. American pubiicT ^ • 

^ I've -proposed several criteria which I think might lie useful to keep in 
m^.nd when considering questions of productivity and effectiveness in a young \ 



'but growing intellectual field. There are indicators of: (1) system growth; 
' (2-) systemic differentiation,; (3) specialization as well as of growljag Leader^- 
^ship and, most importantly;^ (4) increased use of scientific standards of <iuali 
ty for knowledge production. These criteria are related both ,t9 the initial 
goals and objectives of federal involvement in educational r & d and to how 
we think about and characterize growth and development in a. number of domains\ 
I'm sure- others could add to this list and will want to. 
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P, Michael Timpane 
Deputy Director 
National Institutje of Education ' 
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Washfhgton, D»C, 

I found this a devilish difficult topic to deal with and I would like to 
just say a few words about it at two levels. One level is how we run our 
procurement policy; in other Words, how through pur techniques and procedures 
we try to make the research and' development effort which we are purchasing more 
productive. The thrust of. that will .really be to confess the limitations on< 
our leverage, on any such control perspective, because I think that is indeed 
tl\e moral of my little story. And then IM like to talk very briefly and 
generally about what some paths to real productivity in r & d might be. ^ 

A sl^nple definition of p#bductivity is output per unit cost — that seems 
to be what the economists call productivity — and that^s the measure of producti- 
vity that gets converted into what we call benefit costs and cost effectiveness 
equations, when we move into broader evaluated frameworks. But it seems to me 
that between the notion of productivity and the notion of research and develop-^ 
ment there is really a fundamental contradiction, in r & d to some extent by 
definition output is unknown; and if the output were known, it wouldn^t be a' 
research question. We are all seeking for an answer that isn't there. And so 
while we are searching we use all sorts of proxies like how many reports there » 
will be and how many sections each report will Kave^ and how many' people will ^ 
Work on^the report and so on; those are all really proxies — and fairly weak 
proxies at that — for any direct ^asure of output. Axd output is defined as 
some question answered or some practice improved. Y^t the procurement system 
under which we operate has the fundamental implication that productivity can 
be achieved. There are questions of efficiency and effectiveness that the 
procurement- process can and should deal with to get government more "for its 
dAlar" but I think the question In r & d that is ultimately unresolved is, 
"tftore what?^' * • \ ' 

I don't see any trend in federal procurement policies but that there be ^ 
greater demands for productivity. , Certainly in HBW this is the case. A month 
ago a large-scale pifocurement conference was dialled at the 'Secretary *s behest' 
in which not just beleaguered contracting officers, pr even beleaguered deputy 
administrators, but beleaguered agency heads sat around for a day or two being 
— w6 won!t say indoctrinated — but advised of the- requirements of procurement ^ 
poli'fey. We have, an HEW circular coming out with .more vigorous contracting . 
requirements: more clout for procurement officers; more requirements to identi- 
fy what the monitoring^ requirements will be before the contract is awarded; 
what the site ^vislt schedule will be; how quickly the program officer must- re- 
port any^^slippage to the contracting .^^^icer; how many days after the. completion 
of the contract will the program officej: write a full report to the contracting 
officer as to the fulfillment of the'^fequirements. , 



The* 'Whole procurement process bottomed, ^s I said, on the notion of 
productivity. This, emphajsis on productivity is making the use of those 
procurement instruments ^nore and more difficult in the area of research and 
development.^ At that conference^ my job was to tell them that at the very . 
least in' working up procurement regulations, they should have some separate ^ 
sections aind separate requir'ements for the purchase of- research and develop- 
ment , as* opposed to the purchase of pencils and the rental of large buildings, 
and so on, that may occur. I must say- there is still a considerable reltictance 
to do it. It probably would require something that does not exist in the 
federal government, and- that- is an entire cadre of procurement- of ficers whose ^ 
specialization and career lie in the area of research Such officers are 
rare indeed and for that system to operate well, it would require mdny more i 
than presently exist. - . . , * 

Another trend, I. think, will be toward more contracts in general. I mean- 
the underlying logic would be toward f ^.xed price contracts .rather than cost 
plus. This is particularly 'Important for NIE because NIE historically has 
sought productivity through the procurement process. The results of a report * 
that the staff recently did in respqnse to some of the Campbell committee ^recom-' 
mendations to our National Council,* showed that NIE has relied a great deal 
on the procurement process as its path to productivity in r & d. Between -70 
and 85 percent of our e:tpenditures in the past several fiscal years have gone . 
out in form of contracts, either competitive or sole source. Last ysar, by - 
way of a sort ordinal of other numbers, we put out something like 35 majpr 
RFP's including not just those to your organizations but other majo'r competitive 
procurements," and we ran three grants" competitions in an unsolicited proposal 
process. So you can see w^ere the weight of NIE's efforts has lain; And it 
is much more so than other research agencies. That was another poiht this re- 
port makes and one chat we have just begun to reflect on. I think it is going 
to take us a little more time, to face up to its implications — th^ this whole 
procurement thrust . may be in some important respects i4 not a blln^ alley, a 
shott street*^ that may not take us too far toward productivity. S 

The Campbell report was really quite good on that — this process — and in 
spelling out its- limitations. I will just read some of tl;iese sentences from 
the Campbell report because they pretty much reflect. some of the observations 
I would have made after just a few months at the agency.^ "It ^is not always 
true that th6 agency staff can write clear and useful specifications. for what 
is wanted." True. 

"The costs of bidding are eventually added to the_ government's costs of 
future procurements ." True. « • ^ . 

"When there are only a few good^^p^rformejrs for a given type of work, the 
rest of the competitors have little chance and their costs of f^ljure are a 
drain oon energy and time 'that might be avoided." , 



* R&D Fundirig Policies of the National Institute of Education: Review and 
Rec ommend a t ion s , Final Report of Consultants to the Director and the National 
Council on Educational Research, Roald F. Campbell, Chairman^ (Washington, D.C.: 
^National Institute of Education, August 1975).. 
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On productivity: "Extensive competitibn among a small number of organi- 
zations capable of 'large-scale work in educational r & d may tend to- promote 
^isentegration and secrecy." 

"Where proposals are judged by interested staff, the current procedure 
prevents them' from working with proposers, to loo*k at ideas, or to review 
advance copies of proposals so as to avoid submission of obviously unresponsive 
or unqualified ones." That's interesting jto pause and comment about a bit. 
For some reason I don't fully understand, NIE and HEW as a whole are under 
greater constraints in that area than are some other federal agencies. It 
appears from this report that NASA and quite a few other agencies have for a 
long time had extensive adyance discussions and its been all quite copestic.^ 
But it's quite clearly constrained severeTy'if not tjj^ally prevented by HEW 
regulations. « 

So, those, I think, are some pretty interesting observations on the limita- 
tions of procurement process, the process upon which NIE has relied probably 
most heavily. Now, what is the procedural 'alternative? The procedural alter- 
native is the grant. As far as I can make out, other — various in betweens — " 
have been proposed. ^ * . 

The grant, in theory, offers some advantages over the contract in terms 
of productivity of r .& d. If the peer review process is thorough and of high 
quality, and if ^he agency runs an intelligent grant process oyer a series df 
^ears, I would say that that way ought to be the patji to productivity* The 
real. path ^o productivity is in a vigorous t & d system, not £n any control 
efforts that* we are likely to be able to effectuate. ^ 

It turns ouL that in HEW the process of making grants is very complicated. 
"When we go to do a .gratit competition in, say, basic skills (which is one are 
dping now) we -have to go through the identical procedures that the Office, of 
Education has to go through to put out i£s regulations, on Title I. Under the 
HEW regulations, a grant is a grant is a grant. That it is a grant to a state 
or locality to run a program or a grant to an individual researcher to perform^ 
a project, is not well'^distingulshed at all In the HEW regulations. So in the 
ease of the basic skills grant, dt took us, I belieb/e, 18 months from > the be- 
ginning of trying tp*^put out a grant proposal until 
Once again this does not appear to be truet-in every 

generalized regulation for grants competition apfd it^iins all its grants 
compe,titipns, year in and year out, under *the same regulations — puts out the • 
little notice saying *Nthis year or thi3 month, the 'following is. our topic and 
away it goes and out goes the money. Again, for reasons I am just beginning 
to explore and understand, HEW — not just NIE — but HEW is^not" in that position 
and I think that is an issue we have got to worry about.* - 

So that puts us in a real*pickle in terms of,,».tlijB instruments we havQ 
available and the controls we can put on them if the contractural process uses 
up enormous amounts of our time. It is based on thi** assumption that we can 
specify all kinds of things that we probably can't always specify veify care- 
fully in terms of performance of research* And the grant process is so slow 
Moving and dif f icurts-tfnat we have the unproductive situation pf waiting a year 



we got it out the doot. 
other agency. NSF has a 



jbr so before being able to do anything, once we have decided to give some grants. 
iSo I think we are under some constraints and I really think that wjiile we .could 
make that process more efficient, the constraints are really rather normative. 
After all, whil^ we are a mission ageni^y, whilft we do heed to give out ,our 
resources according to, certain priorities that Congress has set upoii us, it^is 
research and development that we are dealing with* We are not going to have 
the resources nor the wisdom to bring about much Increase in productivity iti 
r~&,d just by pulling the few strings we have. I think to some extent we have, 
tried that and to some extent it hasn't worked. 

^ * * 

Now we have begun to try; to shift patterns of our procurement . a little bit 

' to the extent that we have instruments available. I think that the longer tenn 
contractural arrangement such as is envisioned in the process that we are all 
going through now, is probably in some cases an instrxanent which offers a. little 
more realistic management of educational r & d.than does the short term RFP. I' 
think that with all their limitations, some of which I hope we can get relaxed,, 
grants do offer more possibilities than we have realized so far. I think, for 
example, that we have never really been genuine in our opening up oqr grants 
competition. We have neyer really had the. kind of trust itf the field that a 
grant competition implies. We have never made peer review more than advisory^ " , 
in either grants or contracts, and while we have to proceed with care in this • 
area, I think that some expansion of grants competition is probably 9ne path 
we will take to at least try to incjpease productivity by building the health - 
of the system, if you will, rather than through control mechanisms. This is not 

* to say that we won't be responsible for the lawful and efficient expenditure of 
federal funds, and the day will never come when you just get yoi^r checks in the 
mail as from Social Security. ^ ' ^ 

I think th^t we are going to be searching for a balaflce,^ but that balance 
will be constrained because none of the instruments available are that terrifi- 
cally grekt on enhancing r & d productivity ^in the short run. So that^ leaves 
me at the end with just a general kind of statement which is that productivity 
in ah r & d system, at least from a» federal agency perspective', will be manifested 
more or less in the overall health and vigor of that system. X think that, in 
general, we have to approach productivity indirectly and In terms of the whole 
-NIE program rather than directly and through the performance of specific projects. 
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I guess my task is to talk about how tlje Det>artment of Defense, in general, 
and the Nav^ Department, in particular, atteii?>t to ensure productivity and ef- 
^ fectiveness in educational r & d. However, I have to give you the obligatory' 
disclaimer that my reniarks\are^'unof ficial and do not reflect official policy 
of either DOD or the Department of the NaVy. 

y - - , 

From what I*Ve heard, it seems that DOD's funding strategy for our DT&E 
is maybe in a little better shape than some of the other federal agencies* 
funding strategies. In particular there lare various categories of money wjiich 
are budgetsftd for from basic research, on the one hand, to transition, to opera- . 
tional use on 1:he otrher hand.* Some of you may or may not be familiar with this, 
but there are more or less six categdries of money that can be applied to this 
process beginning with independent laboratory research through /exploratory 
development*, advance development, engineering development, management, and sup- 
port up to operational system "developmettt. I think the leg-up that this gives 
us is that there is explicit funding for the transition of products to the 
users . . 

; , . ' • ^ , . ^ ' . 

'DOD runs its work in a couple <of ways^ There are various/^agencies with DOD 
that fund basic research and some of the more advanced developmen^'^inds of 
things. The main ones are the Defense Advance Research* Projects Agency and 
then in the Navy, the Office of TJ^val Research. The organizations that are 
responsible for more of the advance development of ^products that are likely A:o , 
be more applicable to particular ser^ce problems are my organization. Navy 
Personnel Research and Development Center,' the" army *s organization which is the 
Army, Research Institute, -and the Air Force Human Resources Laborat*ory. ' y / 

- I^ you think a little' l?lt' about it, the military does an"* awfully good job 
in training. it*s a huge traiiiing estabrishment and dui:;ing peacetime can be 
' likened to a large junior college. The Navy, for example, has 'something like . 
5,000 different training programs, courses, or schools for which it is tesponsi-^ 
ble, and its problems are very serious. They include the declining manpower 
pool; the increasing technological complexity of all of the weapons. systeBis, 
propulsion systems, arid so forth; 'and particularly the** decreasing dollar sup- * . 
port fqr i^anpower and personnel. , * * ' ' 

The military alao does a reasonably gootl job of transitioning r & d to opera- 
tional use. Jtost of the developments in educational research have had ^their 
history. Kind* of their birth in the military. This began primarily durihg and 
after World War II with the psycholofeical sciences branch of O&R and other 
agencies like that, and resulted in products like job task and content analysis , 
systems approaches to training. The Navy, for example, right now runs the 
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world's largest computer and its instruction system. So they have been reason-r 
ably up on thipg^. - ^ ^ ^ 

Like most of the other government agencies, the work that is done out-of- 
house i-s done mainly on contract and not much on grants'. And as most of you 
know, those k^ds.^of things Are competitive procurem«nts, sole source procure- 
ments, and unsolicited ptoposals which are also sole source.. Now, atour 
particu^lar organization, given th^ problems that Mr. Timpane has mentioned in ^ 
the procurement proceiss, there i^^ot a whole lot that we. can do to maintain 
productivity or effectiveness because when the government is letting a contract, 
it is buying a product, and unless it is very careful, it can't do much about 
speci^ing how that product is to be developed. As far as insuring effective- 
ness is concerned, our organization — we are primarily* concerned with applica- 
tion of products to Navy .pi:obleras, especially j^n the latter funding categories 
(the advance development and things like that) — is a little better off than some ^ 
others, in that we 'fund about half uf our budget in out-of-house work anci the 
rest is in-house. . So, -for example, I monitor a couple, of contracts but I also 
do a lot of research in-house, so 1 am on both sides of it. And w^^^^e con- 
tinually required to^ justify everything upstairs, ju^ like everyone else is. 

Anyway, ajLl of* our work is fairly programmatic and, therefore^* we are un- 
likely to fundi anything that doesn't contribute In some way to the programs 
th^t we are interested -in at the time. The main concern is th# usability of ^ 
utility of the products resulting from our r & d and, therefore, the efforts 
that we fund are -going to have to have a high pjrobability of resulting in a 
product which may be applied in Navy educatipn, or training, or manpower, or 
something. • ' o • 

Once the Work is contracted, abo^t the only recourse" we h^^^^is to monitor 
^closely. We attempt to ensure Navy application and that means tl^, when possi- 
ble ,t the work should concentrate on Navy problems. Navy subject matter. Navy 
research subjects or students. Al^o, hecau^e we have an in-house staff, we do 
most of the follow-on work, including the; prototype test and evaluation, and 
then transition to the operational users. % 

' ■ ■ ■ t ' ^ 

Now one of^the things that has been laid on up- in the past year or even 
before that, is the requirement for cost/benefits analysis on almost everything 
that comes out. We are required to specify the implementation costs and all 
the attendant costs, t ha t wo u 1 d be in vo 1 ve d in imp 1 emen t ing s ome inno va t io n in 
education and training systems. As far as productivity goes, about the only 
things that we can do are attendant with' the contract ^process during procurement, 
during the initial evaluation of a proposal or during. the specification of an 
RFP.*^ We have t6 be very carefql about how we specify what the contract is for. 
We can assess the ability of proposed. contractors to complete the work as pro-, 
posed or a^ requested and that's one of the criteria with which procurement 
decisions are made. We can assess the reasonableness of cost^ of completing 
work, although that'fe not ordinarily done by the technical people; that's done 
by the contract people and those two processes are separate for competitive^ 
procurements in particular.. And then in the contract, all we can do is require 
frequent progress reports, data items, and descriptions describing how, the 
work is progressing and what's happening. And, finally, we can include require- . 

jnents for review and approval of final products before the contract is completed.- 

<», ' — - 
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We have, as I mentioned earlier, the same problems from our perspective as 
researchers: we ^re continually required to demonstrate that we have been 
productive* One of the things that happened recently was the^GAO got more or 
less "slcced" on Department of Defense humah resources r & d about a year and 
a half ago* So the GAO came through and looked at all of the DOD r & d labs* 
Those people are nialnly accountants and they like to count things, so they then 
wen^ to users of our research and asked them if they Kad ever heard ^of "oifr 
research or used it* They came back with the finding that about 30 or AO per- 
cent was not tised, and they seemed to be somewhat upset about it* I was kind 
of gratified that, if you turn it around the other way, 60 or 70 percent of 
it was used* 
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^ U.S. Department of Justi'ce * 
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We have somewhat the same problems^as the educational 'people who are work- 
ing on a .social problem, - so to speak.' I thJ^nk we are probably under the sdme / 
guii right now f roitf a federal standpoint as many of the othex. social programs in 
the United States government. We, too, have to account for*actioM. We h^ve 
many different problems than .you might find in the DOD; in fact we .looked at 
the DOD mpdel in constructing the Institute eigiit years ago and one thing we found, 
different from DOD was that our buyers are not within our own organization. 
Therefore, the decision to go into basic research or into applied research or 
to implfement, was among many differ^t parties, «o this coiiiplicates the proh^- 
J.em of decisions all along the process. In response to Mr. Timpane's message; ^ 
I would think that all federal agencies right now are being asked to clean up 
, their internal procedures. We, too, are being asked to go more towards the 
contract mechanism but-we've had somewhat "more liberty, perhaps, than HEW, be- 
ing an organization ^which is much newer and much smaller in stature and fund-, 
ing. 

We have outlined a very new procedure with respect to basic research We 
will fund all of our basic research using concept papers, program announcements, 
and the grant mechaijism. The grant. mechanism would be a virtual gift to the 
grarftee, as comparedtb a contract where we would expect something to be delivered 
to the United States government, perhaps a set of punch cards or a JElnal report, 
something like that. With a grant, we wouldn't expect anything/^occur: we 
would expect at this particular time" to determine exactly how nmch of our research 
money goes into basic and applied re^e^ch. - ^ ^ 

We have just undergone review by four major institutl^'^i^t' tlie Hopse Com- - 
mittee on Science and Technology and the Judiclatyhon^ir^tee ot the House; the 
•American Bar Association T the National Academy of SMeri^ea^j^ and ^a bepartmetit 
of Justice' study which was initiat^ soon after Caxp^ cUme into office. The 
consensus that we cap, see — and we have just com^jpted this analysis — is that we 
may go down the same path as the National Ins^iltut^ of Education;/ it appears 
that the Congress, ^the National Academy of Sciei^es,' and eyiejryoni' who has looked 
at us over the last 8 years ,^ would desire much mor^e indepefidenc^on- our part. 
Essentially we have, been too politicized, according to themi'""'The agency" heads • 
have used our resear'ch money fo.r action programs and by/direction or edict 'or ^' 
Whatever you want to call it. It^ would appear that the Con'gre^s will not 8ucc%b ' 
to any Department ^of Justice move which would do otherwise, so we may become an 
independent agency. ... . ' 

' ' ■ ■ ■ ■■•';./■■-".. ■ 

' . - -62- I 

* • ■ 66 , " 



However,, during the hear|.ngs the same message came through as the subject 
of this particular conference on productivity: What do w^fe get from our re- 
^search? What we are trying — and it's my job as the head /of a new office in 
the Of fic'e of the^ Director — is a new ahaj.ytlc unit to look at the long-range 
.aspects of the National Institute, to answer such questi«nis as productivity - 
. and, to see and hear what other people havje.^to say ±n other fields., Ve^Ti&ve 
just started a five year program (the program announcement is <Qa the street ^ 
righjfe now for anybody who might be interested in it) to try* to define the 
performance measures of the criminal justice ^system so we can get some kind of 
an idea as to where we are making improvements. If* we can find out when our 
research is aimed %t an improvement, we can possibly measure the benefit and 
the cost of establishing such an improvement. - ^ ^ 

We have s^en, for instance, data that is published by the U.S. Bureau of 
Census on the private sector — all the various industries, the j>rQ4uctivity that 
they have, theik costs of employees, their wage l)ills , salaries bills, v^lue 
added, the number\of shipments, the value of the shipments, o and things like 
this — and we believ^that the criminal justice system can be an econometric ^ 
model similar to thatM^irough which at ledst one age^ncy compares* itself to 
another agency,, from a re3>t4:V^ standpoint anyway. Exactly what the agency is 
doing for'its money may only be^^^me^ured relative what another agency might . * 
be doinga and getting for the taxpayer>-^e are also going to try to develop 
some Ij^efit and cost standards in our functional area. If we do move more 
to^rd that typ^e of measuring scheme, I thrnk- there is some credibility, inr the 
94:andardization of cost accounting data in the United States government and what 
* one »is ^willing to pay to obtain a particular benefit. ^ We tried a little bit 
about this at the national level but may the best man win in getting the budget. 

We are a very new agency and with respect* to productivity, I fully agree . 
with' a Ipt of^what has been said in the last day with respect to the organi- 
zational aspects of trying to inoi;eaSe productivity through , organization. We* 
ourselves are trying increase research and development capacity in the •United 
States by doing most of oujr research outside of the National Institute. ' I ^ 
was surpris.ed ,to hear--and if someone can confirm thisTnumber — that the number^ 
of research and developers or research people in the field of education may 
only range from 3,000 to 15,000. I don^t know whether that is correct or in- 
correct but it seems to be a very small number.. We- are- aiming at much higher" . ^ 
numbers than that, if we can build t^ capacity. 'Obviously we are going to be 
very inefficient in building fhls\ capacity because we probably can lure people 
in for the time being and then thdy might, find other 'endeavors, ^^would like 
to '^see a lot o^t work done ^Ji the ^rea of Ibng^range capacity building hecausa^I 
thifik that h^s a lot^ to do with increase in productivity of thfe research confitainity, 
whether it be associated with the educational flpld or the field' of crime. 

I might add one other thing that I have heard. I have heard such numbers * 
as grant ef feqjtiveness being around 1-5 percent. In ojher words, out of 100 per- 
cent of the granting ^monies, 15 percent are effective. Now that may or may not. 
be^jcorrect bht it is to me an interesting* measure if it is correctl - I don't 
sep anything very bad about that. I think that is actually higher\jitan\ I would 
estimate it to' be. ' 

One other thing is that I think we are finding a phenomerla in the Carter 
administration that we don't quite understand yet, at least a^t the agency level: 



the minimal use of consuitants and advisory, bodies t8 the federa^ agencies. We - 
don't Jcnow exactly ^at that means. We h^^e been stripped of quite a few advi- 
sory bodies In our own agency. Wis have been alldwed to keep our advisory body 
to the National' Institute. With respect to consultants, I don't know what it 
means. Does 'that mean that we should do moke research in-hbuse, because we. 
use many consultants *to evaluate research,^ to procure very small things which - 
are Associated with research? We don't know what the impact of that particu*- 
lar thing is. We thint that It could af'fect our efficiency and productivity 
'of pur research. * ^ ' . 
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The Gyeat Experiment ^ > . - . ' 

* Sin^e World War II our gov^iHiment has been engaged 'In a great experiment 
► in the .use of pyocurement ^to perfoVm r & d (research and development) tasks ' 
Prior to the war, 90 percent of this work was done in gove.mment arsenals and - 
like in's^tallatiQns* Then, ^during the war, governgjent^and indus-try joined forces^ 
in a mammoth r & d effort to produce the atom bomb. The success^ df l:his~-enter-r 
prise undoubtedly served to entourage the development -of *the contract system. 
As a result, whole new industries laeie established, new technologies wer,e inr 
vented and technological development was generated at the fastest pace in the 
hd^story o/ man^ ^ . . ' ' , 



rebver; this activity stimulated a revolution in« the field of management,. 
Under the auspices of the Deparment of Defense co^tractd^ -management program's, 
PERT )f^^rogram Evaluation and Refview Technique) PPP (Pfaa^^ Procurement Plan- • 
ning), and PPBS (Plannirig, Programming, liudgeting Systems) ,^ were developed. 
A large-scale, decentralized r & d operation,- that many governments would have 
been reluctant to attempt, performed successfully. The Ultimate test was* the. 
moon landing, which was the product of the effiprts of a government, agency plus 
tens of thousands of contractors. ^ The United States assigned world leadership 
in r & d. The contract/grant system became a. proven way of promoting innovar- 
tion through the collaboration of government / industry, jiniversities , attd noij^ 
profit organizations.*^ Although this system was not without its faults,'^it 
prove^. to be a reasonably good fit to th^ requirements' of the r7& d tas^. i 

^ The agency^contractor system offered flexibility in programming, directing, 
managing infqrmation flows, and in the* use .of resources. There was the poten-- *^ 
tiaL for the test of the market and the •qtiality of output to serve as regulatqrs 
of activity. The ''activlty-drient^d" permanent ^government bur^auct&cy was . re- 
placed by a joint enterprise that was "achievemfent-oriented" and flexible. 
When objectives were no.t*being realized, it was simplex:^ to close down a Dyi^asoar 
program and lay xytf Boeing workers than it would c have been to shutdblPh a . govern- 
ment fadility employing 'career civil service pejr^onnel. ^ 
" • ' . • ♦ ^ ' -A- . .J 

The^^ system also provided a good fit with certain of our economic artd cul- * 
tural values. We developed^ a ^;type^of federalism by cont^ract. JThe contract <- 
aystem is congruent with the political judgment that the best government is 
spmll government. "With- this system the government is able to i^ndertake a large 
amQunt of activity without a c^^incomit^nt growth In federal government. 

JProcureioent rather than" government in-house activity b^ca^ie. the'^ vehicle for 
promoting effective r & d. *In the process a number of distinctive appj'oaches 
to the agency-contractor relationship were developed. * ^ . 



* j 

Grants programs for r d involve the greatest freedom for recipients, a 
freedom which they deem absolutely essential"! Jn initiating the grant-awarding 
process, an agency announces support for wo^k ijx a given area and -requests 
proposals. The proposals submit t^d^-are then ey4luat-^d by review committees 
composed of peers of the proposal waiters. [ Thi-s process stresses research merit^ 
and if the process Is fuxictioning well, slipshod 'research will be screened out 
and high quality ptojects will be selected. | * The hoped-for output of the 'sys- 
tem is two-fold: advances in science and technology and the development and^ 
strengthening of recipient institutions. ' Agency- initiated controls are minimal* 
or nonexistent* because for a lohg time the ^orld of federal grants tto profit 
and nonprofit research institutions, universities, and hospitals was thought 
;to be self-regulatin§. 

Contracts differ from grants in a number of ways. In the first place, con 
tracts involve r & d work closely allied to and relevant for an agency's heeds. - 
In the second plaqe, thiS' relationship has never been regarded as totally self- ' 
regulating. In fact, controls are almost universaj.ly regarded as- essential. 

. , - ' . i ^ . - = • ■ 

In a\l cases the contract imposes substiantial constraints on the relation- 
ship. The document specifies time limits arid other conditions ^f or completion ^ • 
of the assignment. The' natyre of the produqt may be precisely defined, and * 
final payment awaits delivery. Financial C?introls are an important feature i 
They vary all the way from the comforts -of '^cost plug" and generous progress^ 
payments to a stringent "fixed price" model. In addition, a seemingly endless 
variety of economic incentive systems has been developed, some so c^splex'^as to 
defy implementation. Given the .uncertainties in the r d process, agreement 
on appropriate controls has been slow in coming, ^and as .a result, this aspect 
of the agency-contfractor relationship ha3 beeji the subject 6t much experimentation. 

Cultural values have favored a market-^pe model as the ideal for the 
agency-contractor relationship, but in practice thfe constraints involvjed in doing 
business with the government have led to the establishment of of^anizations or > 
units within organizations that specialize in this particular work. Thus, a 
.dependency-type "off-market" .relationship has developed in , many cases. ' * 

The technological characteristics of a particular agency's work impose fur- 
ther constraints on the relationship. The differences between the Department 
ot Defense (DOD) and the National Aeronautics and Space Agency (HASA) are a case 
in point. '^DOD'^s missile and aircraft development programs,^ with their large 
follow-on production orders,- permit a relationship i^ which the agfendy -relies 
heavily on the contractor for technological^ expertise and on systems techniques 
for control. NASAj^.on the other hand, produces ^one-of-a-kind spacecraft and 
cannot toleratie errors in their design and production.' NASA thus employs a 
system for close monitoring pf contractors supported by an in-house capability 
sufficient to assure the quality of/ the monitoring effort.. 

Certainly^ all parties directly and indirectly involved in this great ex-* 
perlment would agree that there is no one best way , to conduct^ the* r & d agency- 
contractor relationship in order to achieve effective performance. ^The demands 
of the environment, the task, the nature of* the individuals and organizations 
^involved require organizat;ional arrangements tailoted to f it^, each particiilar . 
case. . - " . * , - * ' < 



- No one claims that" the style of r & d contracting developed in -the hay-, 
d^y of the 1950s and the L960s' was idedl. Several congressional committees -raised 
issues such as cost overruns and systematic biases in agency-contractor rela- 
tionships. HoVever, in making critical analyses, one can become so 'obsessed, 
with finding faults that one fails^ to pbseTve the strengths of the system. Too 
often one then proceeds' tp focus reforms* on the curbing of faults rather than on 
.attempts to build on strengths • - '\ . 

• * 

• Re ce^feHTe ve lopmen t^ 

\ ' * * * . ' 1* . 

Recent developments indicate -that the past as we haye known it will not 
continue. We will set forth new pressures for change and then proceed to 

• examine their likely' impact on the agentry-contractpr relationship and on the 

^ effectiveness o£ the r & d» produced by it: Is the^stateof the present admit- 

tedly Imperfect fits between the r & d task and org^izational* arrangements 
'-likely to "be. worsened >D;?--dm§rov§d? - * ^ 

* y ' 

Within the pasif 5 years^at deast two ma^or changes have tiaken place in thfe 
^ field ^ of federal t & d 'grants and contract?. The first is t he ec ytWmic crunch 

which has caused the agencies to search for "fat" in their budgets • Unfortu- 
, nately, a high proportion (85 percent in some cases) of these budgets represent 

. fixed obligations. R&D grants and contracts are one of the few source's of 
/ flexibility. In addition^ ^when budget cuts threaten staff members* jobs^ 

contracted out r & d tasks provide a sgurce of work that can 6e pulled inside, 
" and civil service unions have been pressuring for this course of action. In 
the process, funding for externally conducted r & d ha^ not only been reduced, 
it also has become unstable. The contrajctor who relies on a steady flow of ' ♦ 
federal grants and- contracts is faced with an uncertain future. , 

The second change involves a shift in emphasis. Given i:?ie ampler funds of 
the 1950p and 196bs, innovation 'in weapons and §pace technology pushed the state 
of the; art in these fields far beyond thdt which prevailed in the civilian world. 
This self-generating innovation' led to* an Imbalance in the' econojjy and fostered 
pressures to equalize by Investing more heavily In social programs. While 
space and defense are far from being put out of business,. Health, Education, and 
Welfare (HEW) is the rising star and social programs have become the big growth 
area ih federal spending iEor r & d. However, in part because of, the pressures 
described above and in part because the technology, involveH is less complex, 

HEW has been less inclined tp contract out rid." ^ ^ ^ * 

^ ' ■* ^ * ' * , • 

As a result of ; the .above trends, the government has moved away from jelying 
• • on contractors as heavily ,as it once did. In ;1960, 91 , percent of "all r & d y 
,was contracted out, while in 1976,. the proportion wis '72 percent. By way of 
exaptple,' In 1977 'ffiW* contracted out only one-^fourth' of the 2 billion -clollars ^ . 
It spent on program evaluation research. \ ^ . ^ ^ 

. . » Today we, are faced- with what the contractor can only i regard as an unenvi- 
able -situation: 'less activity and less stable funding com^bined wilth greater 
'f pressures, on and criticism of the r & d Work thatTls being funded. In 1977 
much publicity .was ^jiven to federally "supported r d blunders and scandals. 
A highly critical Senator William Proxmire named the Department of Transportar 
tiion the AuguQ^t \tlnn^ of his Golden Fleece, o^ the Month Award' for a ^225,0,00 

\ / * ' • , • -67- - , , ' ' \. 
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contract with the Stanford Research Institute to forecast transportatJLon needs 
in the year 2025 urid^r hjrpothetical' conditions that included guerilla warfare ^ 
arvd the establishment of a dictatotship . The following finding waf .reported: 
•"'In the ev€nt- of guerilla warfare^ automobile use in affected regions would 
become risky. "-^ Readers, who have no access to information that would verify 
.whether or not the above quote fairly represents the work in question, are 
easily led to tlie conclusion that all government-supported social science r & d 
is a waste of money if not an outright fraud. 

^ <^ " ^ « 

Late in September 1977, there arps« cries of* s'Andal concerning .r & d funds 
management. The pooling of funds from various grants Is a, long-time pr^actice of 
institutions engaged in government-supported r & d. The practice developed 
because both funders and contractors have difficulty arranging the flow of funds 
to meet a project's immediate needs. An agency may supply funds months after 
work on ^.prbject has' already- commence^, tn the meantime; to me^t expenses the^ 
contractor uses funds received for work on other projects. This constant jugging 
of monies represents an accommojdation to real'life pressures. 'A project begins 
with a proper budgg^t, and in the final accounting the proper amounts will have*" 
been expended in each category, but in between there exists a condition that may 
be chara^cterised either as "needed leeway" or as J'iaw breaking." 

0 * ' ^ ) 

In the case in point, a young arss is tan t professor at Harvard alleged, that * 
he had been dire(jted -to sign ,Blank forms vouching for the way in which his 
National Institutes of Health ([NIH) grant monies had been spent. The Depart- 
ment of Nutrition at the School of Public* Health* then 'filled them in with unre- 
^iated items and forwarded them to the government. NIH found that these ^allega- 
tions wete true and at .the same time uncovered serious accounting problems, in 
two oth§r^ grants. Harvard was asked to pay .back $132,349 for misspentting »on aj.1^ 
the grants, and an audit wa's begun* "on' the tqtal of. $400 mij-lion in federal- fundus * 
that the school" "receives. , ' ^ 



Agency Activism 



. Clearly, the. pressure is ofi.^ Bureaucrats are being told to "straighten but" 
the situation in r, & d. The issues that have surfaced are partly effectiveness 
and partly control, with the* former serving as the object of rhetoric and the 
la^^er as the object of action plans. , ^ - 



-• What has been happening in response to the pressures? Most notably, agjsncy 
'activisnj- is on the upswing, as eviddnjed by 'the following developments.*^^ 



Results (outputs not inpats) are being stressed. Tn the process,' , ^^'^^^ 
government funding is being Accompanied by pressures to do targeted 
applied research rather th^n basic researph wil:K its;, Xong gestation ' 
.periods and rfsk^ that it may ^ot" yield praciiical spinoffs. These^ \ 
pressures are g'fenerated' by demands from mariy In and out o(f politics ^ 
for faster, payoffs from the- investment in federal r & d. A * 

\ . ' " ' ' ^ ' , I ^ . , , , 

Measurement and evaluation of the .results of federally supported " * . 
r jSt d are/being; stressed. New political attitudes demanjl- some kind 
of human justification fpr research. ^Ih part to satisfy this demand, 
output .indicat-ors are being developed. In addition, awards 



,of federal funds are now accompe^nied by a requirement for 
follow-up evaluation research. The allocation of ' funds for 
ex post facto evaluations is growing by leaps and bounds. 

• Accountability is being stressed. More and more is re- 
quired in the way pf time and effort reporting, monthly • n 
certification, detailed justification, guaranteed schedules, 
and affirmative action plans. Some observers maintain that - 
r & d^managers are becoming mere paper shufflers. 

The availability of appropriate management systems has both conttibuted to 
and supported these developments. Planning Programming and 'budgeting Sxstems 
(PPBS), with emphasis on input-output analysis, have provided managerial tools 
for furthering the new era' in r & d controls. PPBS has created' a climate de- 
manding analysis. Some Bureaucrats haveflatched onto PPBS, for they see in it 
a golden opportunity,. Billions of dollar^ are involved in 'meeting^ the demaiRis 
of projected federal PPBS programs, and these programs will create* hundreds *atid 
even thousands of new ca^eer^. . . ^ • ' 

However, PPBS also asks much of the agency that'us^s it. • Questions of 
priorities, comparative Costs, benefits, effectiveness; and resource inputs , 
and outputs are raised routinely. In addition to developing plans. and i)rior- 
ities for its own»work, an agency must coordinate with other agencies in order 
t9 participate in overall r & d planning and policy-making. Some unhappy 
decisions may have*to be faced. A particular 'agency's r & d project may appear 
'to be cost-effective in its own right, but in the overall scheme of things a ' 
more productive use -may appear for the .funj4s this project wjas hQB^.ng to "receive 

A troubling question immediately arises concerning th^se developments: 

Are the critics who deplored the looseness in the old. agency-contractor rela*- • 

tionship about to render r\ d unproductive and ineffective in the process of ^ 

tightening up? The old, ways 'were far from ideal but ^there* was a semblance 

of a^fiC wi'th thfe requijrements of r'&'d work in terms of flexibility. Will 

the new system lead to increasingly bad fits between the X d task' and the ^ 

organizational arrangement^ involved, which in turn could serve ta severely ^ 

diminish' the 'ejuality of the output? 

• • * * '* 

What will be the impact of these developments cn the «agency-con^actor 
relationship'? -The pressures on an agency to come up ^with. a total r' &,d plan;* 
can lead that organization to mdve into* the* contractor 's- realm with force/ to 
the ekten£ that* the contractor may become , lit tie more th4n-a ''body 'shop^'I- 
an extensldti of • the agency, providing the personnel fot its cfl-ostd-y sp*ecifi^*^ 
tasks. ^ ' ' - . 9\ * 

In'additi<^, as the requirements for much pat>erwork and reporting and 
specifically targeted research bgcome well known, it is possible that new a^ta^ 
*of contractors will spring up,* ready and^wllling tfo ^oeet these deinands. They, 
will fulfill immediate requirements, btit iftight not 'the lohgrrijp worth of their* 
contributions be open to question? The agency could lose valuable Contrdcfor * 
inputs in the process of, narrowly specifying outputs and t|jus* severely delim- 
iting the** con^ract;pr*s..rn3.e. /, . * 



r 



In this new planning milieu, vjfhat will happen to some 6f the time-honored 
r & d institutions' designed to assure productivity and effectiveness? ^^e^^ 

.review for project selection. is an example. PPB3 and cost/benefit analysis\ 
perform the identical function. Critics see peer review as nothing more 

.than an "old boys network" in operation, but students^ of the subject nSintain, 
thdt it is the best means for selecting research proposals that will prove to 
be 'productive. Still, the question remains: If peer review loses out, will 
the jpdgments that replace it be of equal quality in terms of the results 
produced? ^ 

Assessment and Analysis . 

Experts in the field of management maintain that effective r & d requires 
flexible organizational arranjjements: informal, decentralized structures that 
permit on*" the—spot decision making in response to new diBcoveries, information, 
or unexpected developments. The task environment .±s highly uncertain. and as a 
result, those working in this field require organizational and decision struc- 
ires that enable them to cope with this uncertainty. 



ti^r 



^ Perhaps an encouraging note is sounded by Pelz, who in his comprehensive 
^ research on r & d personnel found that they were most effective in their work 
,when they fac^ c^hallenges which' provided "creative tensions." ^ Thus, the 
researchers' bV#t work was pr^oduced when they' devoted one-fourth of their 
time- to either administration or teaching.^ It is possible that the, inter- _ 
face witl|P^ agency activists will surface' some of this needed tension, but some- 
how we doubt ft. The tensions created are likely to be overwhelming, for as > 
w^ have noted, a ba*d fit is in the process of being developed between the ^ _ * 
r & d task and^ the organizational arrangements designed to control it. f ^ 

We must, look critically at the implications of systemS analysis. It sug- 
gests that central administrators and their, sta'ff analysts can compare the 
outcomes of different r & d proposals ahd make tradeoffs on the basis of .future- 
. oriented analyses in'order to select the best alternative. This, in ^ urn, im- 
plies a degree- o^ control over rid programs, expenditures and decision making 
which may not be present. * ^ ^ / ^ 

Social prdgram^ r & d, the major growth ^rea, is a case, in point. If we 
want to resolve our social problems ,.on a systems basis^> but through » a dec en-* ^ 
tralized system of. contractor participation, w^ will' somehow hav« to develop ^ 
much rddre- knowledge of these problems and of.un'eans for coping with tjheffl. 
Desiifed outputs can then' be specified aijd results measured and evaltiated. 'If this 
knowledge is*lacking, and we still want to proceed with contracting on ,a systems 
basis, then we ^ill bfe obliged to surround the r '& d process with regulations, 
rules, limits, standard contracts,* "boiler plate," and rigid spetif ication of 
inputs. To some extent; at least, this is .a ^descfciption- of the present situation. 

Where the* federal government .is .concerned, systems politics and process 
politicg will' always co-fexist. Systems politics focuseS"*on results and outqomeS 
^ and^not on what, is done to get them. Process politics involves the interaction 
of interest groups and ,the striking of bargains that represent^not^ a desired 
out come,'* 'but "the best we can get, given the, situation." 



If systems politics denies the outcomes that a group of program consti- 
tuents or r & d conttactors seek, it is very lilcely' that' they will take the 
process politics royte to achieve their objectives. The systems manager whd 
'feels that he/she is the only decision force for a given program is living 
in a dream Hprld. In the Social program area, especially, the interaction 
between systems politics and process politics can be very intense. Public 
services and jobs .are involved. PPBS originated in the Department of De- 
fense, and it is likely that defense programs are much more amenable to sys- ' 
tems' controls than are those involving the civilian world. * 

Jf we are to avoid vast control mechanisms and centralized agency regula-. 
tion of federally supported r & d, we need to have appropriate pressurea and 
incentives built into the programs that provide support for r & df 'This ±s 
the great value and strength of the free market system. If we unwittingly 
inc^ntivize cost overruns, we will get them. If we in^itute stringent 
conti:ols aimed at overruns, we may create a system that overpowers and en- 
feebles the basic r' & d activity. .We must leafn to incentivi'ze the behaviors 
we desire. • ^ , ^ ^ ^ , 

Taking a cue from the free market model, excellence in r & d can be in- 
*duced by designing a self-forcing (for excellence) , self-enforcing (for con- 
trol) system that includes both agency and contractors. To achieve this goal, 
a pressure/incentive system must be devised that will function to correct signi- 
ficant errors and prevent major, distort j^ons from arising.* Relying heavily on 
ind^^ect means, the agency designs, a sys,tem that provides pressures in the 
right ' direction so that most of the time the system will stay on course. '^The 
"x:hoice of a strategy, is a function £>f various factors such as the technology 
involved. Therfe is no one best strategy. , ' \ • 



In general, the agency will wSit to inpure that some of the basic condi- 
tions for effective r & d are preseSp. - For instance, needed information ex- 
changes must take place and problems must surface and be identified almost as 
soon as tfiey ajrise.* $ome me^ns must exist for following problems until they 
have in fact been resolved. 

♦ 

For example, the greater the interdependence among^ the units (contractors) 
' involved in an r & d project, the greater is ^tl)e probability that cooperative 
work relationships will develop, and thus, a 'self-forcing system will emerge. 
' Contracturally^imposed requirements are .of little use in buildirfg such pressures, 
for there is no' way* of defining^in adv^jioe the amount and quality of inter- 
change needed between two organizations tifet must cooperate to assure success"ful ^ 
completion of a given project. If two r & d contractors are apt to have to work' 
together at spme point to resolve shared probleijis, it is advisable to avoid the . 
selection of two organizations that normally compete. with one another. Rather^, 
one should sel6ct two organizations that will continue to need one another *s , 
good will in the future. ^ . ^ ; . ' • - 

The ex post facto evaluation rgsearch so popular in PPBS provides informa-^ 
tion about problems after, they haVe become a part of history. However, to. in- 
duce excellence, the r & d system needs to be incentivized to surface problems 
as they arise. "Problems then can be dealt with effectively ana out in the 4jpen,^ 
quickly enough to take timely^action and to^ ensure that all those ^jj^ally and . 



Mr 



potentially. afiffected will have the information they require.^ NASA's r & "d -pro- 
-grams required this type of visibility, and after the Apollo 'fire,, the agency 
introduced the use of a data bank to routinely enter all unresolvel^ problems 
relating tor a particular program. This system was implemented with the aid of 
a skillful systems integration contractor. It j^roved effective because as a 
top manager put it,' he did not have to issue ordets to a delinquent contractor: 
"Organization A knows it, Organization B knows it. Organization C knows it, • 
I know it, and the offender knows we all know it. He feels compelled to take 
action^'? " ^ i ' ' ^ * 

\ I ^ ^ , -\ ^ 

.Incentives that properly g\iide what is done in the course of a program or 
project are equally or even more^ important than those attached to the end product, 
which is, after all, the result of the activity which preceded it. 

The agency-cont.racjtor system ^as proven to be an effective means fdr bring- 
ing together a variety of talents and resources to attack the r &^d problems 
out country needs to solve. Building inc^tives and pressures into our r & d 
programs is one means for assuring effective performance. We should pursue 
experimentation in this direction. 

In the opinion of rhis writer, itj is simply too early to "lock into" sys- 
tems (PPBS) approaches with their strong emphasis on outputs. We need a broader* 
base of knowledge, and expertise, especiall]^ in the field of social programs, 
.before we move to an approach of this type. As we are not now in a position to 
initiate a well-planned controlled r & d system, it seem wiser to concentrate 
on improving the effectiveness of_ the agency-contractor rela*tionship that has ^ «■ 
served us so well*; - - , * . * ^ * 
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